Analysis finds Google AI Overviews is wrong 10 percent of the time

In today's digital landscape, searching for information on Google often brings users face-to-face with AI Overviews, an AI-driven feature powered by the Gemini model. Since its debut in 2024, AI Overviews has faced criticism over its inconsistent accuracy, although it shows signs of improvement. A recent analysis by the New York Times has shed light on its performance, revealing that while the AI provides accurate information 90 percent of the time, it also delivers incorrect answers one in every ten queries. This discrepancy translates to hundreds of thousands of inaccurate responses being generated every minute, raising concerns for Google. The analysis was conducted in collaboration with a startup named Oumi, which specializes in AI model development. Oumi utilized AI tools to evaluate AI Overviews using the SimpleQA assessment, a widely recognized test designed to gauge the factual accuracy of generative models like Gemini. SimpleQA, launched by OpenAI in 2024, comprises over 4,000 verifiable questions that can be posed to AI systems. Oumi began its testing last year when Gemini 2.5 was the latest model, achieving an accuracy rate of 85 percent. After the release of the Gemini 3 update, AI Overviews improved to answer 91 percent of the questions correctly. However, when applying this error rate across all Google searches, the implications are significant, with tens of millions of incorrect answers potentially being issued daily. The report highlights specific instances where AI Overviews faltered. For example, when asked about the date Bob Marley’s former residence became a museum, AI Overviews referenced three sources, two of which failed to mention the date. The third, Wikipedia, presented conflicting years, and the AI selected the incorrect one. In another case, when queried about Yo-Yo Ma's induction into the classical music hall of fame, AI Overviews misrepresented the existence of the hall despite citing the relevant organization's website.