Cloudflare's latest AI rankings expose the web's biggest free rider

Every quarter, Cloudflare updates its analysis of how major AI companies interact with web content, and the latest findings highlight a troubling trend of excessive content scraping. During the week of July 1 through July 7, the data revealed that Anthropic stands out as a significant outlier, with its bots crawling web pages approximately 2,800 times for every referral sent back. Trailing Anthropic, OpenAI and Perplexity showed lesser activity, while Microsoft and Google followed them with even lower referral ratios. Interestingly, DuckDuckGo appears to maintain a more equitable approach, managing a ratio of three scrapes for each referral back to content creators, according to Cloudflare's metrics. While Anthropic has made some strides from an early April ratio of around 8,800-to-1, the recent data still indicates a high level of scraping. For instance, in the first week of May, their bots scraped websites an astonishing 24,700 times for each referral. These metrics, provided by Cloudflare, serve as a crucial indicator of whether AI companies are upholding or disrupting the historical economic agreement that allowed free crawling of content in exchange for user traffic. In the evolving landscape of generative AI, this traditional deal is increasingly compromised. AI-driven answer engines and chatbots provide users with direct answers, decreasing the likelihood of visits to the original content sources. This shift diminishes the financial incentives for creators to produce high-quality material online. Although Anthropic brands itself as a leader in ethical AI practices, these statistics raise questions about the ethical implications of its own content usage. The company has also criticized competitors for allegedly utilizing its AI outputs to enhance their own models, labeling these practices as "distillation attacks" that violate its terms of service. However, this criticism mirrors Anthropic's own behavior towards web content, suggesting a broader issue within the industry regarding content ownership and usage rights. As the digital landscape continues to evolve, Anthropic may need to reconcile its stance with the realities of content sourcing and the ethical considerations that accompany it.