In Harvard study, AI offered more accurate diagnoses than emergency room doctors

A groundbreaking study has explored the effectiveness of large language models in various medical scenarios, including real-life emergency room cases. Remarkably, one AI model demonstrated accuracy that surpassed that of human doctors. This research, conducted by a team of physicians and computer scientists at Harvard Medical School and Beth Israel Deaconess Medical Center, was published this week in the journal Science. The researchers carried out a series of experiments to evaluate how OpenAI's models compared to the diagnoses made by human physicians. In a pivotal experiment involving 76 patients at the Beth Israel emergency room, the diagnoses from two attending physicians were compared with those generated by OpenAI's models, o1 and 4o. To ensure an unbiased assessment, two additional attending physicians evaluated the diagnoses without knowing their origins. The findings revealed that, at critical diagnostic junctures, the o1 model either matched or outperformed the human doctors. Notably, the model's accuracy was particularly striking during the initial ER triage, where rapid decision-making is essential and information is often limited. The o1 model provided exact or nearly accurate diagnoses in 67% of triage cases, while one physician achieved accuracy 55% of the time and the other 50%. Arjun Manrai, the head of an AI lab at Harvard Medical School and a lead author of the study, emphasized that the AI model excelled against various benchmarks, exceeding both prior models and the performance of human physicians. However, the researchers were careful to clarify that the study does not imply that AI is prepared to handle life-or-death decisions in emergency situations. Instead, it highlights the urgent necessity for further trials to assess the use of these technologies in practical patient care environments. Additionally, the researchers pointed out that their examination was limited to text-based data, acknowledging that existing studies indicate current AI models may struggle with reasoning over non-text inputs. Adam Rodman, a doctor at Beth Israel and another lead author, cautioned that there is currently no formal system in place to ensure accountability for AI-generated diagnoses. He noted that patients still prefer human guidance for critical life-and-death decisions and complex treatment choices.