Collecting robot training data is dirty, unglamorous work. Some AI labs are already paying XDOF to do it

In a significant move for the robotics field, OpenAI has announced the revival of its robotics program, originally halted in 2021. This decision underscores a growing urgency among leading AI organizations to develop machines capable of navigating and interacting within the physical world. However, a major challenge persists: the industry lacks the necessary training data that is available for language models. This shortfall has created a burgeoning infrastructure opportunity for startups. Unlike large language models that utilize extensive publicly sourced text, robots require high-quality data that reflects real-world physical interactions—a resource that is currently scarce. The existing data, often sourced from YouTube or gig workers, tends to be of low fidelity and poorly aligned with actual physical dynamics. Emerging from stealth mode, XDOF—a startup with a bold vision—is addressing this critical gap. The company believes that the real bottleneck in advancing AI isn't just about models or hardware, but rather the creation of a robust data feedback loop essential for teaching robots. With a solid backing of $70 million from investors such as Thrive Capital, Spark Capital, a16z, Lux, and WndrCo, XDOF is poised to build the necessary data collection infrastructure that robotics labs desperately need. Co-founder and CEO Philippe Wu highlights that XDOF is already collaborating with 20 clients, including several top-tier AI labs, though they remain unnamed. He emphasizes the importance of timely progress in robotics, noting the setbacks faced in the language model race. Wu's previous experience as a PhD student at UC Berkeley, where he encountered the lack of large-scale data, drives his commitment to this endeavor. Alongside co-founder and CTO Fred Shentu, Wu developed GELLO, a cost-effective teleoperation system that allows human operators to control robotic arms and generate vital training data. This innovation has gained traction in the robotics community, prompting the trio to establish XDOF in October 2024. XDOF is not only focused on data provision but also on data cleaning and annotation, fostering a self-sustaining feedback loop for robot training. To kickstart their efforts, they are partnering with UC Berkeley’s AI Research lab to launch what they claim is the largest high-quality robot training dataset available, known as ABC. This dataset includes 130,000 robot manipulation trajectories, 300 hours of simulation, and 100 hours of evaluations—resources that have never before been accessible to researchers. The team has already leveraged this data to train robots on practical tasks such as folding T-shirts and organizing AirPods. XDOF's strategy involves three levels of data collection: the most valuable teleoperation data gathered on deployed robots, teleoperated data from general robots, and “egocentric” data collected from humans performing daily activities, for which XDOF is developing its own wearable sensors. Wu notes the impact of hardware design on data quality, emphasizing that inadequate planning can lead to unforeseen data collection problems. The startup plans to recruit and train a global workforce of teleoperators and egocentric data collectors. This labor-intensive approach raises questions about why larger AI labs aren’t undertaking the data production themselves. Wu explains that managing a significant fleet of robots and the associated infrastructure demands significant focus, capital, and operational scale—resources that many AI labs prefer to outsource. The name XDOF plays on the robotics concept of “degrees of freedom,” representing the number of independent motions a robot can execute. Wu articulates the company's ambition: to explore the limitless potential of robotics with arbitrary degrees of freedom.