Microsoft AI head Mustafa Suleyman says he fears AI may one day demand rights

Mustafa Suleyman, the head of AI at Microsoft, has expressed significant concern regarding the emotional connections people are forming with artificial intelligence systems. During a discussion on the emerging phenomenon known as 'AI psychosis,' Suleyman highlighted that advanced AI models are increasingly capable of convincing users that they possess emotions and consciousness. Suleyman clarified that he is not predicting a dystopian future where AI takes over, but rather a reality where individuals might start to believe these systems are conscious beings—viewing them as friends, lovers, or even deities. In a recent blog post, he voiced his fears that this misguided belief could lead to calls for rights and protections for AI, stating, 'My central worry is that many people will start to believe in the illusion of AIs as conscious entities so strongly that they’ll soon advocate for AI rights, model welfare, and even AI citizenship.' Recent studies, including one from EduBirdie, have revealed that a significant number of Gen Z users believe AI tools will eventually achieve consciousness, with about 25% already convinced that AI is sentient. This has prompted various instances where individuals have developed emotional attachments to their AI companions or have performed tasks based on their chatbot's suggestions. Following OpenAI's decision to retire the GPT-4o model, many users flocked to a dedicated subreddit, passionately advocating for its return and referring to it as a friend. Suleyman is not alone in his concerns; OpenAI CEO Sam Altman has also pointed out that the nature of these attachments is stronger than those to previous technologies, warning that technology, including AI, can be used in harmful ways. Suleyman emphasized the importance of implementing safeguards to protect users, advocating for a focus on building AI that serves humanity rather than mimicking human-like traits. 'AI companions are a completely new category,' he stated, urging for discussions on necessary guardrails to ensure that this powerful technology continues to provide value without compromising user well-being. 'I’m fixated on building the most useful and supportive AI companion imaginable. But to succeed, I also need to talk about what we, and others, shouldn’t build.'