Anthropic and OpenAI are crawling the web even more and not giving much back

The ongoing debate surrounding the practices of major AI companies like Anthropic and OpenAI has intensified, raising questions about their impact on the web—a critical repository of human knowledge. Recent findings from Cloudflare have brought to light the extent to which these companies are extracting information from online sources without adequately compensating them. As tech firms invest heavily in infrastructure and talent, the conversation often sidelines a crucial element of AI development: data acquisition. Rather than paying for high-quality human data necessary for training and refining AI models, these companies tend to deploy bots to harvest information from various websites free of charge. Historically, a mutual understanding existed where content creators allowed access to their data in exchange for web traffic referrals, thereby generating revenue through advertising and subscriptions. However, with the advent of generative AI, this agreement seems to be deteriorating. AI-driven platforms now provide users with direct responses, reducing the likelihood that they will visit the original sources of information. Cloudflare began monitoring the interactions between AI bots and websites in early 2025, focusing on the ratio of bot requests to actual referrals sent to websites. This crawl-to-refer ratio serves as a metric to evaluate how much value these tech giants extract versus what they return. For instance, Cloudflare's data indicates that Anthropic's crawling efforts have significantly outpaced its referral traffic, highlighting a concerning trend in early January compared to September 2025. Similarly, OpenAI has also seen a decline in its crawl-to-refer ratio, suggesting it is now taking more from the internet while giving less back to content creators. Reports from late 2024 corroborate these findings, revealing that excessive crawling by these companies has led to increased operational costs for website owners, impacting their bottom lines. Despite inquiries regarding their crawling practices, Anthropic has not provided a clear response and previously cast doubt on the Cloudflare methodology. The company did mention the introduction of a web search feature for its Claude AI chatbot, claiming it would enhance referral traffic to websites. OpenAI has similarly remained silent on the matter. It is important to note that the crawl-to-refer ratio only accounts for web activities and does not include interactions from native apps, which could alter the ratios for some companies. Google, for example, maintains a lower ratio, likely due to its established search engine model that continues to direct users to website links, though it is gradually incorporating AI responses into its search results. As this situation unfolds, Business Insider plans to monitor these metrics closely to observe how AI companies adapt their practices and their potential implications for the web ecosystem.