Waymo says it built a better benchmark for comparing robotaxis to humans

Waymo has announced the development of a sophisticated computer model aimed at answering a crucial question: How does its autonomous driving technology compare to human capabilities? The Alphabet-owned company collaborated with TU Delft to create this model, which they detailed in a research publication released in Nature Communications. This innovative model is expected to provide more precise evaluations than its predecessor, which had been in use for several years. It utilizes a framework known as active inference, which posits that drivers continuously envision potential future scenarios and make decisions to navigate towards the safest outcomes. According to Waymo, this model will enhance their understanding of human behavior in critical situations that their robotaxis may encounter. “In the automotive sector, physical and virtual crash dummies have long been used to assess vehicle safety features,” Waymo noted in a blog post. “Our new model advances this concept, establishing a behavioral benchmark for autonomous driving systems that accurately reflects how a careful and skilled human driver would react in traffic conflicts.” This improved understanding of human driving behavior is essential for autonomous vehicle companies, especially as they strive to evaluate the performance of their robotaxis in accident scenarios. The timing of this development is significant for Waymo, which is expanding its operations into more urban areas and facing increasing scrutiny from regulators and the public. For instance, in January, a Waymo robotaxi was involved in an incident with a child near a school in Santa Monica, California. At that time, the company relied on its earlier model to assert that a vigilant human driver would have impacted the child at approximately 14 miles per hour. In reality, the robotaxi struck the child while traveling at just 6 miles per hour after slowing down from 17 miles per hour, resulting in minor injuries for the child. This incident is still under investigation by the National Highway Traffic Safety Administration and the National Transportation Safety Board. The key enhancement in this new model, designated as the Reference Driver, is its ability to simulate human behavior leading up to a potential crash. Unlike previous models that primarily focused on “last-second, reactive” actions of human drivers, the Reference Driver can emulate the internal “surprise” that a driver experiences during critical situations. Arkady Zgonnikov, an assistant professor at TU Delft, emphasized that this new approach offers a more realistic benchmark for autonomous driving systems, which was previously unattainable at scale. Waymo asserts that the Reference Driver can be customized to represent a wide array of road user behaviors beyond just avoiding collisions. It is designed to efficiently analyze complex real-world crash scenarios in a virtual environment, leading to rapid identification of performance improvements. Additionally, Waymo is encouraging collaboration on this model by making the research code available under an academic, non-commercial license for research, teaching, personal experimentation, and scientific publication purposes.