Twitter’s former Trust & Safety head details the challenges facing decentralized social platforms

Yoel Roth, the former head of Twitter’s Trust and Safety department, is raising alarms about the challenges that decentralized social networks face in regulating harmful content. Now working at Match, Roth recently shared his perspective on the future of the open social web during an interview, expressing concern over its ability to effectively manage misinformation, spam, and illegal materials such as child sexual abuse content. In a discussion on the podcast revolution.social with @Rabble, Roth highlighted the limitations of moderation tools available to platforms in the fediverse, which includes apps like Mastodon, Threads, and Bluesky. He recalled pivotal moments from his time at Twitter, including the controversial ban of former President Trump and the rampant misinformation propagated by Russian bot farms, drawing attention to how even high-profile users like Twitter's CEO Jack Dorsey were misled by bots. Roth emphasized that while initiatives are underway to create more democratically governed online communities, they often lack the necessary resources for effective moderation. He criticized the trend where platforms providing community control simultaneously offer the least robust technical tools for policy enforcement. "Many services leaning into community management have been left with inadequate mechanisms to uphold their governance standards," he remarked. He lamented a significant regression in transparency and legitimacy within the open social web compared to Twitter's practices when it was under his leadership. Roth noted that whereas Twitter provided rationales for its decisions, many modern platforms now avoid transparency to prevent manipulation by bad actors. Users on these decentralized platforms often find their posts mysteriously vanish without explanation, undermining accountability and user trust. Roth raised concerns about the financial sustainability of moderation within these federated systems. He pointed out the challenges faced by organizations like the Independent Federated Trust & Safety (IFTAS), which aimed to develop moderation tools but ultimately had to scale back due to funding issues. He explained, "The economics of this federated approach to trust and safety have never really added up, and they still don’t." In contrast, Bluesky has opted to employ moderators while allowing users to customize their moderation preferences. Roth acknowledged their efforts as a step in the right direction but cautioned that as Bluesky continues to decentralize, it will face challenges in balancing individual protections with community needs. Additionally, Roth discussed the tension between privacy and moderation, noting that while Twitter collected minimal data to avoid unnecessary privacy violations, it still retained key information that aided in identifying malicious activity. He warned that without adequate data collection, it becomes challenging to distinguish between genuine users and bots, as evidenced by his experience at Twitter where users would often label opponents as bots without justification. A particularly pressing topic was the impact of artificial intelligence on social media dynamics. Roth referenced recent Stanford research indicating that large language models can be more persuasive than humans in political contexts, underscoring the need for platforms to analyze behavioral signals alongside content. "Focusing solely on content puts you in an arms race with advanced AI models, which is a battle you are likely to lose," he stated. He advocated for a more nuanced approach to moderation, one that considers user behavior patterns as critical indicators of potential manipulation.