One of legal's hottest startups is helping lawyers finally answer: Is the AI's work any good?

Crosby, a unique blend of a startup and a law firm, is stepping into the spotlight with its groundbreaking tool, Redline Bench, which is designed to evaluate the performance of artificial intelligence in legal tasks. This innovative tool aims to determine how well AI can handle real-world legal challenges, particularly in contract review. As AI continues to advance in various fields, including software development, legal tech companies are racing to harness its potential for tasks like contract analysis, risk identification, and negotiation. However, as Crosby's founder Ryan Daniels points out, defining success in legal work is far more complex than in coding, where outcomes are binary—code either runs or fails. In law, the interpretation of what constitutes a 'good' contract edit can vary significantly from one lawyer to another, creating a challenging landscape for automation. Crosby has assembled a tactical unit called Crosby Intelligence, comprising engineers and legal experts, to create a benchmark for assessing AI performance. This team includes Sharan Ramjee, who has experience in developing fraud detection models at Stripe, and Ross Weiser, a lawyer with a background at the prestigious law firm Sullivan & Cromwell. Their mission is to establish clear criteria for what constitutes effective legal work through the development of weighted benchmarks. To create these benchmarks, senior legal professionals simulated software negotiations and identified crucial contract modifications at different stages. These insights were then transformed into a structured scoring system. When testing AI models, Crosby presents them with standardized contracts and evaluates their edits against the established criteria. A panel of judges assesses these edits to determine how closely they align with the priorities of experienced lawyers. The release of Redline Bench will allow various AI labs to benchmark their models against Crosby's standards, with the first results already revealing that ChatGPT 5.5 performed the best, achieving a score of 50.5%. Following closely were Gemini 3.5 Flash and Claude Opus 4.8, with scores of 45.1% and 44.4%, respectively. Crosby also conducted a test on Anthropic's new model, Fable 5, which scored 47.3% before it was withdrawn from availability. While Crosby is not alone in its quest to measure AI capabilities in legal contexts—companies like Harvey and major AI labs also release their own performance benchmarks—the reliability of these results can be questionable, according to Daniels. As AI technology reshapes the legal industry, trust will be paramount. Crosby’s initiative seeks to provide the legal community with the confidence needed to adopt these new tools, as billions in investments hinge on the promise of AI-driven efficiencies in legal work.