Cognition's CEO says token spend leaderboards are 'directionally correct' — but some people get carried away

In a recent episode of the 'Founders' podcast, Scott Wu, the CEO of Cognition, shared his insights on the use of token spend leaderboards in the AI sector, suggesting that while the concept may have merit, its implementation often misses the mark. Wu expressed concern that some companies have overly focused on how many tokens employees are spending rather than the actual output being generated. "It is directionally correct, but there are certainly instances where people have taken it too far," Wu stated. He emphasized the importance of measuring performance based on tangible results, such as the volume of work completed, rather than solely on token usage. Founded in 2023 by Wu alongside Steven Hao and Walden Yan, Cognition is known for its innovative AI software engineer, Devin. The San Francisco-based tech company has gained significant attention and investment, raising over $1 billion earlier this year, which has positioned it as one of the leading AI startups globally. During the podcast, Wu acknowledged the high costs associated with computational resources but noted that if engineers can enhance their productivity threefold through the use of AI, it justifies the expense. However, he cautioned that firms should link rewards to more concrete metrics, such as project efficiency and problem resolution. Wu is not alone in his critique of the token-based reward system. Jacob Lauritzen, CTO of legal AI startup Legora, previously voiced similar concerns, stating that merely incentivizing token usage can lead to detrimental practices like 'tokenmaxxing', where employees waste resources to inflate their standing. Andrew Feldman, CEO of Cerebras Systems, further criticized the notion of unlimited tokens for employees, calling it a poorly conceived idea and advocating for a more resource-conscious approach. He likened the efficient use of resources to shopping at Costco, suggesting that a more strategic approach to AI tools could yield better results without unnecessary expenditure.