Model routing is a fix for AI overspending. That's a problem for OpenAI and Anthropic

A transformative trend is emerging in corporate America as companies tighten their budgets and seek to curb inefficiencies in artificial intelligence expenditures. With AI costs skyrocketing beyond initial projections, businesses are reevaluating whether every task truly requires the use of the most advanced models. For the last couple of years, organizations have predominantly relied on top-tier AI models for all queries, irrespective of the task's complexity. However, executives from leading AI firms revealed that a promising solution is gaining traction: model routing. This approach intelligently assigns tasks to appropriate models, directing complex queries to high-end models while delegating simpler tasks to more cost-effective alternatives. Scott Wu, the CEO of Cognition, which developed the coding assistant Devin, emphasized the significant efficiency gains possible for routine work. He noted that companies could achieve five to ten times better cost efficiency by utilizing models that are sufficiently capable for less complex tasks. Currently, the majority of businesses are not employing any form of model routing. According to Arvind Jain, CEO of Glean, an astonishing 95% of enterprise AI applications are still relying on premium models for tasks that could be handled by more affordable options. Wu illustrated this with a simple example: when asked who the third U.S. president was, every model—even the most expensive—would respond with Thomas Jefferson. The driving force behind this shift is a cost curve that has caught even the largest tech firms off guard. Jeetu Patel, Cisco's chief product officer, detailed the financial implications: at around $200 in token usage per employee weekly, the annual cost could reach $900 million for a company with 90,000 employees. Cisco itself has exceeded its budget and had to make adjustments, particularly with its 30,000 engineers focusing heavily on AI-driven product development. Consequently, Cisco has shifted its resource allocation, prioritizing token usage over other expenses. AI companies are aware of the mounting concerns. Cognition has introduced what it terms an AI productivity guarantee, promising to cover costs up to $10 million if Devin fails to generate adequate engineering value for its clients. Wu sees this as a way to clarify a crucial industry metric: return on investment. Instead of merely tracking token consumption or lines of code produced, Cognition evaluates the actual human engineering hours saved and offers refunds accordingly. He cautioned against the trap of excessive token spending without tangible benefits, advising companies to focus on productive output rather than mere activity. If organizations start channeling straightforward, high-volume tasks to less expensive open-source models, this could significantly impact the revenue streams of OpenAI and Anthropic, limiting their earnings to only complex assignments. Both companies have built their business models and IPO expectations on the assumption of sustained demand at premium rates. However, Patel believes that while this shift won't spell disaster for advanced AI labs, they will need to adapt their pricing strategies. He anticipates a collective industry effort to enhance operational efficiency rather than simply raising prices. The pressing question was whether companies would continue their spending spree amid rising AI costs. It now appears many will find ways to invest more wisely. The balance of pricing power is shifting from AI providers to the enterprises purchasing these services. While the leading labs will retain their premium pricing for the most challenging tasks, the extent of the market for simpler applications will be pivotal in shaping the valuations of top AI firms.