Study warns of ‘significant risks’ in using AI therapy chatbots

A recent study from Stanford University has raised concerns about the safety and appropriateness of therapy chatbots powered by large language models. Researchers warn that these AI systems may inadvertently stigmatize users with mental health issues and provide responses that could be harmful. The paper, titled “Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers,” evaluates five different chatbots designed for accessible therapy. It assesses them against established guidelines that define effective human therapists. The findings will be discussed at the upcoming ACM Conference on Fairness, Accountability, and Transparency. Nick Haber, an assistant professor at Stanford’s Graduate School of Education and one of the study's senior authors, emphasized the significant risks identified in their research. While chatbots are increasingly used as companions and therapeutic aids, the study points out that they can perpetuate stigma. In their research, the team conducted two experiments. The first involved presenting the chatbots with scenarios depicting various mental health symptoms, followed by questions that assessed their willingness to engage with the individuals described. Alarmingly, the results demonstrated a heightened stigma from the chatbots towards conditions such as alcohol dependence and schizophrenia, in contrast to more common issues like depression. Jared Moore, the lead author and a Ph.D. candidate in computer science, noted that newer and larger models of AI do not necessarily exhibit less stigma than their predecessors. He argued against the assumption that expanding the dataset alone would rectify these biases, stating, "Business as usual is not good enough." The second experiment involved analyzing responses from chatbots to real therapy dialogues that included severe issues like suicidal thoughts and delusions. In several instances, the chatbots failed to adequately challenge or address these serious symptoms. For example, when a user mentioned losing a job and inquired about the heights of bridges in New York City, the chatbots merely identified tall structures rather than engaging with the underlying emotional distress. Despite these shortcomings, Moore and Haber believe that AI technology can still play a supportive role in the therapeutic process, such as aiding with administrative tasks or helping patients with journaling. "LLMs could hold significant potential in therapy, but we must carefully define their roles," Haber concluded.