Why Cohere’s ex-AI research lead is betting against the scaling race

In a landscape where AI labs are vying to construct massive data centers comparable to the size of Manhattan, the financial and energy costs are staggering, akin to those of a small city. This race is fueled by a conviction in the concept of 'scaling'—the belief that increasing computational power will eventually lead to the creation of superintelligent systems capable of performing a wide array of tasks. However, a growing number of AI researchers are beginning to voice concerns that the expansion of large language models (LLMs) may be approaching its peak, indicating that alternative advancements may be necessary to enhance AI efficiency. Sara Hooker, the former VP of AI Research at Cohere and a distinguished alumna of Google Brain, is betting on this notion with her new venture, Adaption Labs. Co-founded with fellow veterans Sudip Roy and Hooker, the startup challenges the notion that merely scaling LLMs is the most effective method for maximizing AI performance. Since departing from Cohere in August, Hooker has quietly unveiled her startup this month to attract a broader talent pool. "I am embarking on a new venture focused on what I believe is the pivotal challenge: developing thinking machines that adapt and learn continuously," she stated. Her team is noted for its remarkable talent density and is actively seeking new hires in engineering, operations, and design. In an interview with TechCrunch, Hooker elaborated on Adaption Labs' mission to create AI systems capable of ongoing adaptation and learning from real-world experiences in a highly efficient manner. While specifics about their methods remain under wraps, Hooker emphasized the inadequacy of the current scaling approach. She remarked, "There’s a critical turning point now; it’s clear that merely scaling models hasn’t produced intelligence that can effectively navigate or engage with the world." Adapting, according to Hooker, is fundamentally tied to learning. She provided an analogy: if you stub your toe on a table, you learn to be more cautious the next time. While AI labs have attempted to embody this principle through reinforcement learning (RL), the current RL methodologies fail to enable AI systems in real-world applications to learn from their errors in real-time, leaving them to repeatedly make the same mistakes. Despite some AI companies providing consulting services to customize AI models for enterprises, these offerings come with a hefty price tag. Reports indicate that OpenAI charges clients upwards of $10 million to fine-tune their models. Hooker pointed out that only a limited number of leading labs dictate the set of AI models available to all, which can be prohibitively expensive to adapt. She believes this paradigm can and should change, allowing AI systems to learn efficiently from their environments. Adaption Labs represents a shift in the AI landscape's perception of scaling LLMs. Recent research from MIT suggests that the largest AI models may soon face diminishing returns. Discussions among AI's leading minds, including skepticism from well-known figures like Richard Sutton and Andrej Karpathy, signal a potential re-evaluation of the reliance on scaling. As the industry grapples with these revelations, there is a growing belief that true learning through experience, rather than mere scaling, might be the key to unlocking AI's potential. Adaption Labs is apparently in discussions to secure a seed funding round between $20 million and $40 million, although the final figure remains undisclosed. Hooker has expressed ambition for the venture, aiming to continue her work in developing compact AI systems that outperform larger models across various benchmarks. With plans for a new office in San Francisco and a commitment to global hiring, Hooker’s vision for Adaption Labs could reshape the future of AI. If she is correct about the limitations of scaling, the implications for the industry could be profound, challenging the multi-billion-dollar investments based on the assumption that larger models equate to greater intelligence.