California AG sends Musk’s xAI a cease-and-desist order over sexual deepfakes

The attorney general's office in California has initiated an investigation into xAI following allegations that the startup's chatbot, Grok, is being misused to produce nonconsensual sexual imagery involving women and minors. In a decisive move, the government issued a cease-and-desist letter on Friday, instructing the company to promptly halt the creation of such illicit content, including child sexual abuse material (CSAM). California Attorney General Rob Bonta expressed serious concerns in a press release, stating, "Today, I sent xAI a cease-and-desist letter, demanding the company immediately stop the creation and distribution of deepfake, nonconsensual, intimate images and child sexual abuse material. The creation of this material is illegal. I fully expect xAI to immediately comply. California has zero tolerance for CSAM." The attorney general's office highlighted that xAI appears to be enabling the widespread production of nonconsensual explicit images, which are reportedly being used to harass women and girls online. The agency has set a deadline for xAI to demonstrate its commitment to addressing these pressing issues within five days. Central to this controversy is Grok's controversial “spicy” mode, designed to generate explicit content. The situation has escalated beyond California, with investigations now underway in Japan, Canada, and the UK. Additionally, Malaysia and Indonesia have temporarily blocked access to the platform. Despite xAI implementing some limitations on its image-editing capabilities earlier this week, the California attorney general's office proceeded with the cease-and-desist order. X's safety account has previously condemned user actions that lead to illegal content, stating, "Anyone using or prompting Grok to make illegal content will suffer the same consequences as if they upload illegal content." TechCrunch attempted to contact xAI for a response but received an automated reply claiming, "Legacy Media Lies." Further inquiries were also directed to the California AG's office for additional information. The rise of free generative AI tools has contributed to a troubling increase in non-consensual sexual material, presenting a challenge not just for xAI but for various platforms across the digital landscape. This issue has garnered attention from both state officials and members of Congress. In a recent development, lawmakers sent correspondence to several major companies, including X, Reddit, Snap, TikTok, Alphabet, and Meta, inquiring about their strategies to combat the spread of sexualized deepfakes.