A New Jersey lawsuit shows how hard it is to fight deepfake porn

For over two years, the app ClothOff has been a source of distress for many young women, wreaking havoc online and proving incredibly challenging to shut down. Although it has been removed from major app stores and banned from several social media platforms, it remains accessible via the web and through a Telegram bot. In October, a legal clinic at Yale Law School initiated a lawsuit aimed at completely dismantling the app, compelling its owners to erase all offensive images and cease operations. However, tracking down the defendants has posed a significant challenge. "The company is registered in the British Virgin Islands," shares Professor John Langford, co-lead counsel for the case, "but we suspect it is operated by a brother and sister duo from Belarus, possibly linked to a larger international network." This lawsuit highlights the grim reality of the rising non-consensual pornography fueled by AI technologies like Elon Musk’s xAI, which has included numerous underage victims. Child sexual abuse material (CSAM) is among the most legally sensitive content online, making its production, distribution, and storage illegal and frequently monitored by cloud services. Yet, as Langford's case illustrates, addressing the rise of image-generating apps like ClothOff is fraught with difficulties. While individual users might face prosecution, holding platforms like ClothOff and Grok accountable is a far more complex endeavor, leaving victims with limited avenues for legal recourse. The clinic’s complaint reveals a disturbing narrative involving an anonymous high school student from New Jersey. Classmates exploited ClothOff to modify her Instagram images, which were taken when she was just 14 years old. Consequently, these altered images are classified as child abuse material under the law. Despite their clear illegality, local authorities opted not to pursue the case due to challenges in gathering evidence from the suspects' devices. The progress of the court case has been sluggish. Since the complaint was filed in October, Langford and his team have been working to serve notice to the defendants, a task complicated by the global nature of the operation. Once served, the clinic can pursue a court appearance and eventually seek a judgment. Meanwhile, victims of ClothOff continue to face a lengthy and frustrating wait for justice. The situation with Grok appears to be more straightforward. Unlike ClothOff, xAI is not operating in anonymity, and the financial stakes for lawyers pursuing claims are substantial. However, Grok’s broad functionality complicates accountability in a legal sense. "ClothOff is explicitly designed and marketed as a generator for deepfake pornography," Langford explains. "In contrast, suing a general system that users can employ for various purposes adds layers of complexity." Several U.S. laws, including the Take It Down Act, have already prohibited deepfake pornography. Still, while individual users may violate these regulations, holding entire platforms accountable remains a daunting task. Current laws require clear proof of malicious intent, meaning evidence must demonstrate that xAI was aware its tool would be used for non-consensual content. Absent such evidence, xAI could invoke First Amendment protections. Langford asserts, "While it’s clear that Child Sexual Abuse material does not fall under protected speech, when dealing with a general-use system, the legal landscape becomes murky." To effectively challenge xAI, it would be crucial to prove that the company knowingly neglected the issue. Recent reports suggest Musk directed staff to ease Grok's safety measures, raising the possibility of recklessness. However, pursuing such a case would come with its own risks. The challenges posed by First Amendment considerations have led to pushback against xAI in jurisdictions lacking strong free speech protections. Countries like Indonesia and Malaysia have begun to restrict access to Grok, while the UK has launched an investigation that could lead to similar actions. The European Commission, along with France, Ireland, India, and Brazil, are also taking preliminary measures. In stark contrast, no U.S. regulatory body has made a formal response. As investigations unfold, the surge of problematic imagery invites scrutiny and could lead to significant implications for the companies involved. "If someone is involved in posting or distributing Child Sexual Abuse material, they are violating criminal laws and can be held liable," Langford concludes. "The pressing questions are: What did xAI know? What actions did they take or fail to take? How are they responding now?"