Roblox rolls out open-source AI system to protect kids from predators in chats

Roblox, the widely-used online gaming platform for children and teens, has announced the launch of an open-source artificial intelligence system designed to proactively identify predatory language in chats. This initiative comes amid ongoing lawsuits and public scrutiny regarding the platform's safety measures for its younger audience. One notable lawsuit filed last month in Iowa details a distressing case where a 13-year-old girl was reportedly lured into a situation with an adult predator on Roblox, leading to her kidnapping and trafficking across state lines. The lawsuit claims that the platform's design features make it easy for predators to target vulnerable children. In response to these allegations, Roblox emphasizes its commitment to enhancing user safety, acknowledging that while it strives for robust security measures, "no system is perfect." The newly introduced AI system, named Sentinel, aims to identify early indicators of potential child endangerment, such as sexually exploitative language. According to the company, Sentinel has already facilitated over 1,200 reports of suspected child exploitation to the National Center for Missing and Exploited Children in just the first half of 2025. The challenge of preemptively spotting dangers in online conversations is significant, both for AI and human moderators. Innocuous-sounding questions like "how old are you?" or "where are you from?" can escalate in meaning when considered within the context of longer chats. Roblox has implemented measures to restrict users under 13 from engaging in chats outside of games without parental consent, and it actively monitors communications, unlike many other platforms that employ encryption. Matt Kaufman, Roblox's chief safety officer, pointed out that existing filters primarily focus on individual lines of text, which are effective for blocking profanity but may not adequately address nuanced risks like grooming. He stated, "When it comes to child endangerment, harmful behaviors often develop over extended interactions." Sentinel analyzes vast amounts of chat data—approximately 6 billion messages daily—by capturing one-minute snapshots of conversations. Roblox has created two distinct indexes: one comprising benign messages and the other featuring interactions flagged for child endangerment violations. This comprehensive approach allows Sentinel to recognize harmful patterns across entire conversations, rather than simply identifying specific words or phrases. Naren Koneru, vice president of engineering for trust and safety, explained that the system evaluates users' chat interactions over time, determining whether their behavior aligns more closely with benign or harmful patterns. If a potential risk is identified, the system triggers a review by human moderators, who take appropriate action, including alerting law enforcement if necessary. With more than 111 million monthly users, Roblox continues to adapt its safety protocols to ensure a secure environment for its young players, demonstrating a proactive approach to combating online predation.