An AI chief for one of the world's largest banks says tokenmaxxing is a 'vanity metric'

In the current landscape of Silicon Valley's AI frenzy, the trend of 'tokenmaxxing'—which revolves around maximizing AI usage and token consumption for productivity—has garnered significant attention. However, Charles Holive, the Chief AI Officer at BNP Paribas CIB, is taking a different approach. He believes that focusing solely on token consumption is misleading and prefers to measure actual outcomes instead. Speaking at Mistral AI's summit in Paris, Holive expressed his skepticism towards vanity metrics. "We try to move away from vanity metrics—billions of tokens per day," he explained. His team prioritizes assessing tangible results over mere token usage. Instead of asking employees how many AI tokens they've utilized, he prompts them with questions like, "What did you accomplish that you couldn't before? How much more efficiently did you complete your work?" Holive's perspective comes at a time when various companies in the U.S. are reevaluating the return on investment from their AI initiatives. For example, Amazon has recently dismantled its internal AI leaderboard, following reports of employees performing tasks merely to enhance their ranking. Similarly, Uber's COO has raised concerns about whether the increasing costs of AI are translating into valuable products, while GitHub has shifted Copilot to usage-based pricing amid escalating AI expenses. In his role at BNP Paribas CIB, Holive begins AI projects with clear expectations regarding the potential revenue or productivity they may yield. His team then establishes key performance indicators (KPIs) and tracks their progress regularly. While token consumption is monitored to manage costs, it is not the primary focus of their strategy. "We still have dedicated teams to monitor token consumption because I need to control my costs," Holive noted. Echoing Holive's sentiments, other executives at the summit, such as Amit Kapur from Tata Consultancy Services, emphasized the importance of focusing on business outcomes rather than just token metrics. Antoine Pichot from La Banque Postale added that AI projects should be assessed based on efficiency, customer satisfaction, employee engagement, and overall financial impact. Despite his critical stance on overemphasizing token usage, Holive acknowledged that tracking token consumption can still provide insights into adoption levels, particularly in software engineering contexts. "We didn’t say, 'use the tool, do your best,'" he clarified. "We took the opposite approach."