I asked 4 executives how they measure AI ROI. None started with AI tokens.

In a notable discussion at Mistral AI's recent summit in Paris, four industry leaders shared their insights on measuring the return on investment (ROI) for artificial intelligence. A key observation emerged: none began their evaluations by referencing AI token usage, which has become a contentious topic in the tech community. Charles Holive, the chief AI officer at BNP Paribas CIB, emphasized the importance of focusing on outcomes rather than what he termed 'vanity metrics.' He remarked, "We try to move away from vanity metrics — billions of tokens per day. Our goal is to track outcomes that truly matter." Holive further elaborated on the questions he prefers to ask, such as, "What did you accomplish that you couldn't before? How much faster did you achieve it?" This shift highlights a growing trend among companies prioritizing efficiency and tangible results over mere token consumption. Antoine Pichot, director of innovation, digital and data at La Banque Postale, echoed this sentiment, stating that the bank assesses AI's impact by evaluating employee efficiency, customer service enhancements, and overall value for investment. Amit Kapur, chief AI and transformation officer at Tata Consultancy Services, reinforced that his primary concern is whether AI initiatives are bolstering business performance rather than the sheer volume of tokens consumed. This perspective reflects a broader industry shift. Sujay Bhattacharya, NTT DATA's executive managing director, also highlighted the change in focus among his clients. He noted that organizations are increasingly looking beyond just the number of tokens used and are instead concentrating on the overall costs and business benefits derived from AI projects. This evolution in mindset comes at a time when some U.S. companies are moving away from the concept of 'tokenmaxxing'—the belief that increased AI usage correlates directly with enhanced productivity. For instance, Amazon recently canceled an internal leaderboard tracking AI usage after employees began performing tasks solely to improve their rankings. Uber COO Andrew Macdonald has raised doubts about whether increased spending on AI leads to better products, pointing out the challenges in establishing a clear connection between token usage and customer value. Furthermore, companies like OpenAI, Anthropic, and GitHub are implementing usage-based pricing models, which compel businesses to demonstrate that higher AI engagement translates to significant returns. Despite this shift, executives like Holive maintain that monitoring token usage remains essential for cost control and understanding adoption rates. However, they unanimously agree that while token metrics can indicate usage levels, they fall short of revealing the true ROI of AI investments.