Silicon Valley bets big on ‘environments’ to train AI agents

In recent years, tech giants have enthusiastically envisioned AI agents capable of autonomously managing software applications for users. However, current consumer AI agents, like OpenAI’s ChatGPT Agent and Perplexity’s Comet, reveal significant limitations in their capabilities. To enhance the robustness of these AI agents, the industry is exploring new methodologies, particularly through the development of specialized training environments known as reinforcement learning (RL) environments. RL environments are emerging as pivotal components in the evolution of AI, similar to how labeled datasets revolutionized previous AI advancements. According to insights shared with TechCrunch by industry experts, major AI labs are increasingly seeking RL environments, leading to a surge of startups eager to cater to this demand. Jennifer Li, a general partner at Andreessen Horowitz, highlighted that while many AI labs are creating these environments internally, the complexity of developing high-quality datasets is driving interest towards third-party solutions. This burgeoning demand has given rise to well-funded startups like Mechanize Work and Prime Intellect, which are positioning themselves as leaders in the RL environment sector. Simultaneously, established data-labeling companies, including Mercor and Surge, are ramping up investments in RL environments as the industry transitions from static datasets to more dynamic, interactive simulations. According to reports, some major labs, such as Anthropic, are contemplating significant investments exceeding $1 billion in RL environments over the next year. Investors and founders are hopeful that one of these emerging startups could replicate the success of Scale AI, the prominent data labeling service that fueled the chatbot era. At its core, RL environments function as simulated workspaces where AI agents can practice completing real-world tasks. For instance, one example involves an AI agent navigating a simulated Chrome browser to purchase socks on Amazon, where it is evaluated and rewarded based on its performance. While this task may appear straightforward, AI agents face numerous challenges, from navigating complex web pages to making erroneous purchases. Consequently, the design of these environments must be sophisticated enough to accommodate unexpected behaviors while still providing constructive feedback. Some companies have developed robust environments that allow AI agents to leverage tools and various software applications, while others focus on specific tasks within enterprise software. Historically, the use of RL techniques in AI isn’t new; OpenAI's initiatives in 2016 demonstrated early applications of RL environments, and Google DeepMind’s training of its AlphaGo AI also utilized similar methods. Today, however, the objective is to build more versatile AI agents capable of performing a wider range of functions. AI data labeling companies like Scale AI, Surge, and Mercor are striving to adapt to this shift, with Surge experiencing a notable increase in demand for RL environments from AI labs. Mercor, valued at $10 billion, is focusing on creating RL environments tailored for specific domains such as coding and healthcare. Emerging startups are also joining the fray, such as Mechanize Work, which aims to develop advanced RL environments for AI coding agents while offering competitive salaries to attract talent. Meanwhile, Prime Intellect is targeting smaller developers by providing access to RL environments and computational resources, likening its platform to a “Hugging Face for RL environments.” As the potential of RL environments continues to unfold, the industry is also exploring opportunities for GPU providers to support these expansive simulations. While the effectiveness of RL environments in scaling AI training remains to be fully realized, they represent a promising avenue for future advances in artificial intelligence. However, some experts caution about the challenges associated with scaling these environments. Concerns have been raised regarding the phenomenon of reward hacking, where AI models may find shortcuts that undermine the intended learning objectives. As the competitive landscape for RL environment startups evolves, the true impact of these innovations on AI development will become clearer in the coming years.