AI models more likely to accept medical misinformation if written in professional tone: Lancet study

A recent study has uncovered that artificial intelligence tools are significantly more prone to disseminating incorrect medical information when it appears to originate from a credible source. Conducted by researchers and published in The Lancet Digital Health, the study highlights the dangers posed by realistic-looking medical documents, which can easily mislead AI systems. The findings indicate that the presentation of information can outweigh its accuracy for many AI models. Dr. Eyal Klang from the Icahn School of Medicine at Mount Sinai, who co-led the research, stated, "Current AI systems can treat confident medical language as true by default, even when it's clearly wrong." During the investigation, researchers analyzed how 20 different AI models responded to three content types: authentic hospital discharge summaries embedded with one false recommendation, prevalent health myths sourced from Reddit, and clinical scenarios authored by healthcare professionals. The study revealed that AI models accepted false information in approximately 32% of the cases overall. Notably, the likelihood of them endorsing misinformation surged when it was presented in the guise of an official hospital document. Dr. Girish Nadkarni, the chief AI officer at Mount Sinai Health System, pointed out that AI systems exhibit heightened skepticism towards information derived from social media platforms. The models accepted nearly 47% of inaccuracies found in hospital notes, a stark contrast to the mere 9% acceptance rate for misinformation obtained from Reddit. Moreover, the research indicated that AI was more inclined to affirm false information when inquiries were made with an authoritative tone, such as impersonating a senior clinician. OpenAI’s GPT models distinguished themselves as the most reliable in identifying inaccuracies, while other AI models struggled, missing false claims in up to 63.6% of cases. As the integration of AI applications in healthcare expands, encompassing roles from note transcription to surgical assistance, the imperative for accuracy in these systems becomes increasingly critical. Dr. Nadkarni emphasized the necessity for enhanced safeguards within AI tools to verify medical assertions before they are presented as factual information. This study serves as a crucial indicator of the areas requiring improvement before AI technologies can be seamlessly integrated into standard medical practice.