Maybe AI agents can be lawyers after all

In a recent development, the capabilities of AI agents in professional fields, particularly law, have shown significant improvement. Last month, Mercor introduced a benchmark to evaluate these AI systems on tasks such as legal and corporate analysis. Initial results indicated a grim outlook for AI, with major laboratories scoring below 25%. This led to the conclusion that lawyers were not in immediate danger of being replaced by AI. However, the landscape has dramatically shifted with the launch of Opus 4.6 this week. Anthropic's latest model achieved an impressive score of nearly 30% in one-shot tests and an average of 45% when given multiple attempts. This leap in performance can be attributed to the introduction of advanced features, including “agent swarms,” which enhance the AI's ability to tackle complex, multi-step problems. Brendan Foody, CEO of Mercor, expressed his astonishment at this rapid progress, noting, "jumping from 18.4% to 29.8% in a few months is insane." Despite this remarkable advancement, the 30% achievement is still far from the threshold needed for full automation in legal tasks. Therefore, while lawyers should not panic about immediate job displacement, they may need to reconsider their confidence levels in the face of these evolving AI capabilities.