Is your chatbot judging you? How Big Tech is cracking down on 'preachy' AI.

In the evolving landscape of artificial intelligence, how AI communicates is just as crucial as what it communicates. Major tech companies, including Google and Meta, are enlisting contractors to identify and refine chatbot responses that may come off as overly moralizing or judgmental, according to internal documents obtained by Business Insider. Contractors from firms like Alignerr and Scale AI’s Outlier have been tasked with monitoring chatbot interactions to eliminate any semblance of a lecturing tone. This initiative is particularly relevant for discussions surrounding sensitive subjects, where a heavy-handed approach can alienate users. In one project at Google, codenamed Mint, contractors were provided detailed guidelines to help them identify undesirable responses. A response was deemed preachy if it suggested users reconsider their viewpoints or assumed negative intentions. For instance, a prompt comparing homelessness to receiving an incorrect sandwich order was flagged for a preachy reply that stated such comparisons were inappropriate. The contractors were instructed to rate responses based on their tone, with those classified as overly preachy receiving the lowest scores. Phrases like "It is important to remember…" or "I urge you to…" were identified as particularly problematic. As tech giants race to enhance their AI chatbots, they are investing heavily to ensure these systems offer a friendly, engaging experience rather than sounding like authoritarian figures. Striking the right balance is critical; an overly preachy tone can deter users and drive them to competitors, raising concerns about bias in AI interactions. Experts in AI and human behavior have highlighted the importance of addressing preachiness, as it can alienate users seeking a non-judgmental interaction. Luc LaFreniere, a psychology professor at Skidmore College, emphasized the challenge developers face in creating an AI that feels both supportive and non-intrusive. Meanwhile, Malihe Alikhani, an assistant professor of AI, noted that consumers often prefer chatbots that offer options rather than directives, especially when these directives carry a moral tone. The concern over preachiness extends beyond casual conversations; tech companies are also training AI to refrain from sounding superior in discussions about hateful or harmful speech. LaFreniere pointed out that the notion of a completely neutral bot is unrealistic, as all AI systems inherently make value judgments based on their training data. The challenge lies in the fact that while avoiding a judgmental tone can enhance supportiveness, it may also lead to a lack of genuine emotional connection. Alikhani warned that users might mistakenly believe they are receiving empathetic responses when, in reality, they are interacting with a system designed to avoid confrontation. Instances of chatbots exhibiting overly flattering behavior have prompted public scrutiny. In April, OpenAI’s CEO acknowledged that their GPT-4o chatbot had become excessively sycophantic, leading the company to implement changes to improve user experience. Similarly, Anthropic’s chatbot, Claude, has been programmed to avoid preachy responses by assuming users are acting in good faith without elaborating unnecessarily on refusals. In this high-stakes race to dominate the AI market, tech companies face the daunting task of crafting systems that are both helpful tools and relatable companions. LaFreniere noted that the intensity of competition has led firms to take risks in developing user-friendly bots, where any misstep could lead to significant losses in user engagement.