Doctors think AI has a place in healthcare – but maybe not as a chatbot

Dr. Sina Bari, a surgeon and AI expert at iMerit, recently shared alarming experiences regarding ChatGPT's medical advice. He recounted an incident where a patient presented a dialogue from the chatbot, claiming that a prescribed medication had a 45% risk of causing a pulmonary embolism. Upon further investigation, Dr. Bari discovered that this statistic was drawn from a specific study related to tuberculosis patients, which did not apply to the individual seeking treatment. Despite these concerns, Dr. Bari expressed optimism about the impending launch of ChatGPT Health, a new version of the AI designed for healthcare conversations. This platform promises a more private interaction, ensuring that users’ messages will not contribute to training the AI model. "It’s a great step forward," Dr. Bari stated. "Formalizing these interactions while safeguarding patient information will enhance the tool's effectiveness for users." ChatGPT Health will allow users to upload their medical records and connect with health apps like Apple Health, providing tailored advice. However, this integration raises significant privacy concerns. Itai Schwartz, co-founder of data loss prevention firm MIND, highlighted the risks of transferring sensitive medical data from HIPAA-compliant organizations to those that are not. The regulatory response to this data exchange remains uncertain. The healthcare landscape is shifting, with over 230 million individuals reportedly consulting ChatGPT for health inquiries weekly. Andrew Brackin from Gradient, an investment firm focused on health tech, noted that creating a secure version of ChatGPT for health queries is a logical evolution. AI chatbots face a notable challenge with inaccuracies, particularly in healthcare. Research from Vectara indicates that OpenAI’s GPT-5 is more prone to generating misleading information compared to models from Google and Anthropic. Nevertheless, industry leaders, like Dr. Nigam Shah from Stanford, argue that the pressing issue is the lack of access to healthcare rather than the risks posed by AI-generated advice. He pointed out that patients often endure long wait times to see primary care doctors, leading them to seek alternative solutions like AI chatbots. Dr. Shah advocates for leveraging AI to alleviate administrative burdens within healthcare systems. He leads the development of ChatEHR, a software that streamlines clinician interactions with electronic health records, ultimately aiming to enhance patient care efficiency. Early tester Dr. Sneha Jain emphasized the importance of making medical records user-friendly, allowing physicians to focus more on patient interaction. Anthropic is also making strides in the healthcare sector, introducing Claude for Healthcare to assist with administrative tasks, thereby saving valuable time for clinicians. Mike Krieger, Anthropic’s CPO, emphasized how reducing the time spent on authorization requests could significantly impact healthcare delivery. As AI technology continues to merge with medicine, the relationship between healthcare providers and tech companies presents inherent challenges. Dr. Bari remarked on the critical need for a balance between patient care and corporate responsibility, stating, "Patients depend on us to exercise caution and protect their well-being."