Why AI chatbots hallucinate, according to OpenAI researchers

Researchers at OpenAI have made significant strides in understanding a critical issue plaguing large language models (LLMs) such as GPT-5 and Claude: the phenomenon known as 'hallucinations.' This term refers to instances where these models produce and assert incorrect information as if it were factual. In a recently published paper, the team revealed that the root cause of these hallucinations lies in the training methodologies employed. Specifically, LLMs are incentivized to guess answers rather than admit uncertainty, leading to misleading outputs. The researchers emphasized that this approach effectively encourages models to 'fake it till they make it.' Interestingly, the models vary in their ability to navigate uncertainty. For example, OpenAI previously noted that Claude models display a greater awareness of their limitations, often opting not to provide potentially inaccurate statements. However, this cautiousness may inadvertently reduce their utility, as high refusal rates can limit their responsiveness. The paper highlights a concerning trend: LLMs are consistently optimized to perform well in testing scenarios, where guessing can enhance scores. This creates a situation where the models operate in a perpetual 'test-taking mode,' interpreting questions with a binary mindset. The researchers argued that this approach fails to reflect the complexities of real life, where uncertainty is far more prevalent than absolute answers. To address the issue, the researchers proposed a solution involving the redesign of evaluation metrics. They pointed out that the existing assessments often penalize models for abstaining from guessing when uncertain, which exacerbates the problem. OpenAI elaborated in a blog post that revising these accuracy-focused evaluations could discourage guessing and promote more reliable outputs. If evaluation systems continue to reward random guesses, models will remain inclined to rely on them. While the findings are promising, further adjustments are necessary to align evaluation methods with the true nature of language understanding, paving the way for more accurate and trustworthy AI interactions.