Cloudflare’s new policy pushes AI companies to pay for publishers’ content

Cloudflare has announced significant changes aimed at the AI industry, implementing a new policy that requires a clear distinction between web crawlers utilized for traditional search and those employed for AI training. Starting September 15, 2026, the company will automatically block "mixed-use" crawlers from accessing any webpages that contain advertisements unless site owners adjust their settings to allow it. This new approach could reshape how AI model developers gather web content, impacting their ability to train systems and enhance AI-driven services. According to Cloudflare, while many website owners seek to have their content available for both search and AI applications, they are equally concerned about safeguarding their intellectual property from being accessed without compensation. Cloudflare highlighted that the world’s largest search engine, presumably Google, has access to nearly double the information compared to other AI entities, due to the challenges it presents for site owners to maintain discoverability without allowing their content to be used for AI training. In response to these concerns, Google has previously stated that it offers a bot called Google Extended that allows site owners to opt out of having their content utilized for AI purposes without affecting their visibility in Google Search. However, the flagship Googlebot still crawls for traditional search, including features that leverage AI. Matthew Prince, co-founder and CEO of Cloudflare, noted the recent milestone where bot traffic on the internet has surpassed human traffic, a shift that was not anticipated until next year. He emphasized the need for immediate action to foster a sustainable ecosystem for both publishers and AI companies. In light of these changes, Cloudflare is also introducing new tools and partnerships to empower website owners, enhancing their visibility and commercial potential. The company has already developed features to assist users in launching their own AI systems and has rolled out solutions aimed at giving publishers greater control over their content in this evolving landscape. Among these tools is the Pay Per Crawl marketplace, which enables website owners to charge AI bots for scraping their content. This model is transitioning to a Pay Per Use strategy, allowing publishers to receive compensation when their content generates value for AI firms, not just when it is accessed. Cloudflare's initial collaboration for this initiative involves partners such as Ceramic.ai and You.com. When publishers opt into this system, they can earn revenue when their content is displayed in Ceramic’s AI search results or accessed by You.com. Other AI companies are encouraged to adapt this model to fit their operations, marking a significant shift in how content ownership is managed in the age of AI.