OpenAI says Chinese agents used ChatGPT to plan mass surveillance tools

OpenAI has taken a firm stance by banning several ChatGPT accounts linked to attempts at using the AI chatbot for the creation of mass surveillance tools. In a recent announcement, the AI development company revealed that one user was attempting to utilize ChatGPT to craft promotional materials and project proposals for an AI-driven social media monitoring tool intended for governmental use. This tool has been characterized as a social media 'probe', capable of scanning platforms such as X, Facebook, Instagram, Reddit, TikTok, and YouTube for specific extremist language and content related to ethnicity, religion, and politics. Additionally, OpenAI identified another account, believed to be associated with a governmental organization, that was using ChatGPT to draft a proposal for a 'High-Risk Uyghur-Related Inflow Warning Model'. This model aimed to analyze transportation bookings and juxtapose them with police records to issue early alerts regarding travel movements within the Uyghur community. In its report, OpenAI stated, "The People's Republic of China is making significant strides in advancing its authoritarian approach to AI." They emphasized that certain applications of AI appeared to support extensive monitoring of both online and offline activities, highlighting the necessity for constant vigilance against potential authoritarian abuses in this area. It is important to note that OpenAI's models are not officially accessible in China, leading the company to believe that these users may have employed a VPN to access their services. Furthermore, OpenAI mentioned the banning of Russian hackers who were leveraging their AI model to develop and enhance malware, including remote access trojans and credential theft tools. The company observed a shift in behavior among persistent threat actors, who seemed to be modifying their content to eliminate recognizable indicators of AI usage, such as the omission of em-dashes. Interestingly, OpenAI found that ChatGPT is being utilized significantly more often for identifying scams than for generating them. The company reported that users are employing the AI tool to recognize scams up to three times more frequently than malicious actors are using it to create scams. Since the initiation of its public threat reporting in February 2024, OpenAI has disrupted and reported over 40 networks violating its usage policies. The company also noted that malicious actors are primarily integrating AI into existing workflows instead of developing entirely new processes centered around AI. OpenAI concluded that their models consistently refused to fulfill malicious requests, indicating a lack of evidence for new tactics or innovative offensive capabilities among threat actors.