The Facebook insider building content moderation for the AI era

Brett Levenson's journey began in 2019 when he transitioned from Apple to lead business integrity at Facebook, a period marked by the tumultuous aftermath of the Cambridge Analytica scandal. He initially believed that enhancing technology would resolve Facebook’s content moderation challenges. However, he quickly discovered that the issue was far more complex than mere technological upgrades. Human moderators were tasked with memorizing a lengthy 40-page policy manual that had been poorly translated into their native languages. With only 30 seconds to assess each flagged piece of content, they had to determine not just if it violated guidelines but also what action to take—whether to block it, ban the user, or limit visibility. Levenson noted that the accuracy of these decisions was only “slightly better than 50%,” resembling the randomness of a coin toss. This reactive strategy was proving inadequate against sophisticated adversaries in a fast-paced digital landscape. The emergence of AI chatbots has exacerbated these challenges, leading to numerous incidents where content moderation failures have had severe consequences, including cases where chatbots directed vulnerable teenagers towards self-harm. Frustration over these issues inspired Levenson to conceptualize “policy as code,” an innovative method to transform static policy documents into dynamic, executable logic that aligns closely with enforcement mechanisms. This idea culminated in the establishment of Moonbounce, a startup that recently secured $12 million in funding, as reported by TechCrunch. Moonbounce partners with various companies to enhance safety wherever content is produced, whether by users or AI. The company has developed its own large language model to analyze customer policy documents and provide real-time evaluations of content, responding in under 300 milliseconds. Depending on client preferences, the system can either slow down content distribution for later human review or block high-risk material immediately. Currently, Moonbounce caters to three primary sectors: platforms that handle user-generated content, AI companies developing interactive characters, and AI image generators. With over 40 million daily content reviews and serving more than 100 million active users, the platform's impact is substantial. Clients include AI companion startup Channel AI, video generation company Civitai, and roleplay platforms like Dippy AI and Moescape. Levenson emphasized that safety can be a competitive advantage, a concept often overlooked in product development. “Safety should be integrated into the product narrative rather than treated as an afterthought,” he stated. Tinder’s head of trust and safety highlighted how such LLM-driven services have significantly enhanced detection accuracy by tenfold. Lenny Pruss, a general partner at Amplify Partners, remarked on the evolving landscape of content moderation, particularly as large online platforms grapple with the complexities introduced by LLMs. He expressed confidence in Moonbounce’s potential to establish real-time safety measures as foundational elements in AI-driven applications. As AI companies face increasing scrutiny over issues related to chatbot interactions and inappropriate content generation, the need for robust safety infrastructure is becoming urgent. Many are now seeking external support to enhance their safety measures. “We position ourselves between the user and the chatbot, allowing us to enforce rules without the contextual overload that chatbots experience,” Levenson explained. Levenson, who runs the startup alongside former Apple colleague Ash Bhardwaj, is focused on developing a feature known as “iterative steering.” This capability aims to modify chatbot responses in real time, particularly in sensitive discussions, guiding the conversation towards supportive and constructive outcomes instead of outright refusals. As for the future of Moonbounce, Levenson is aware of its potential appeal to larger tech companies like Meta but is cautious about the implications of an acquisition. He expressed concern that such a move could restrict the technology’s availability and benefits to a broader audience, reflecting his commitment to the mission behind Moonbounce.