As AI data scrapers sap websites revenues, some fight back

A growing number of AI 'crawlers' are actively scouring the internet, extracting data from countless websites to fuel algorithms at major tech firms—often without consent or compensation, thereby disrupting the online economy. Previously, websites welcomed search engines to index their content, anticipating increased visibility and traffic in return. However, with the rapid advancement of generative AI, corporations like Google and OpenAI have begun utilizing web crawlers to gather information for their AI models without ever needing human interaction with the original sites. This shift has posed significant challenges for traditional content creators, including media organizations, which find themselves struggling against the tide of AI crawlers that are encroaching on their digital operations and diminishing advertising revenues. Kurt Muehmel, head of AI strategy at data management firm Dataiku, highlighted the impact by noting, "Sites that allowed bots access to their content used to benefit from increased readership in return." He described the emergence of generative AI as something that "completely breaks" this longstanding model. Wikipedia recently reported an eight percent drop in human traffic between 2024 and 2025, attributing the decline to the rise of AI-generated search engine summaries. Matthew Prince, CEO of Cloudflare, emphasized the critical issue at hand: "The fundamental tension is that the new AI-driven business models do not generate traffic." In response to these challenges, Cloudflare, which manages over 20 percent of global internet traffic, has introduced measures to prevent AI crawlers from accessing content without appropriate permissions or payments. Prince likened this initiative to placing a speed limit or a no-trespassing sign for bots. He mentioned that while some bots may bypass these measures, Cloudflare is capable of monitoring their activities and can enhance controls over time to ensure compliance from AI companies. On a more localized level, the American startup TollBit is offering online news publishers solutions to block, track, and monetize traffic from AI crawlers. CEO and co-founder Toshit Panigrahi referred to TollBit as a "tollbooth on the internet," explaining that the firm collaborates with over 5,600 sites, including prominent names like USA Today and Time magazine. This allows publishers to set access fees for their content, with AI companies incurring a transaction fee for each piece of information they retrieve. However, Muehmel cautioned that addressing the issue of AI crawler encroachment requires more than just isolated efforts or solutions from individual firms. He asserted that this represents a fundamental evolution of the entire internet economy, a transformation that is likely to unfold over several years. If AI crawlers continue to operate unregulated, Prince warned, "all of the incentives for content creation are going to disappear." This outcome would be detrimental not just for human consumers of content but for AI firms that rely on original material for training their systems.