Cloudflare wants Google to change its AI search crawling. Google likely won’t.

Cloudflare has recently begun testing innovative features aimed at empowering websites to block AI crawlers or impose fees for data scraping. However, this initiative has raised significant questions regarding its implementation, especially among website owners and SEO professionals. The core concern revolves around how Cloudflare intends to prevent Google's bots from scraping content necessary for AI summaries, without simultaneously hindering Google's ability to crawl sites for essential search engine indexing. This dilemma came to light when a travel blogger questioned the feasibility of the proposed blocking and pay-per-crawl features. In response to the inquiries, Cloudflare CEO Matthew Prince took to X (formerly Twitter) to address the concerns. He asserted, "We will get Google to provide ways to block Answer Box and AI Overview, without blocking classic search indexing, as well," emphasizing his belief that this goal is achievable. When pressed on whether such a separation of functionalities was possible, Prince remained optimistic, stating, "it is. #staytuned." Further engaging with a search engine optimization expert, Prince indicated that Cloudflare is in promising discussions with Google. He expressed hope that these talks would lead to a resolution that allows Google to adapt its crawling methods to function more effectively within Cloudflare's framework. However, should these negotiations fail to yield results, Prince revealed that Cloudflare is considering legislative action as a potential solution. He described this approach as a "very viable option" in various jurisdictions, stating, "Worst case we’ll pass a law somewhere that requires them to break out their crawlers and then announce all routes to their crawlers from there. And that wouldn’t be hard. But I’m hopeful it won’t need to come to that."