Cloudflare and GoDaddy team up to help websites fend off Big Tech's AI bot swarm

In a strategic move to safeguard the interests of website owners, Cloudflare and GoDaddy have announced a partnership aimed at addressing the challenges posed by AI bots. This collaboration seeks to tackle the disruption of traditional online business models caused by the rise of artificial intelligence. As part of the initiative, GoDaddy will incorporate Cloudflare's AI Crawl Control into its web hosting services. This integration will empower website owners to manage how AI bots interact with their content. Site administrators will have the option to grant access to AI crawlers, block them, or even charge for entry, thereby reclaiming some control over their digital assets. The need for this collaboration arises from a growing concern: AI entities from major corporations are increasingly crawling the web to gather content at no cost, utilizing it to generate responses for chatbots and other AI applications. Traditionally, the online ecosystem has operated on a straightforward model where search engines index content, users engage with links, and publishers monetize through ads, subscriptions, or sales. However, this framework is under threat as AI-driven answer engines frequently deliver information directly to users, often bypassing the original content sources, which has led to a decline in traffic and revenue for many publishers. Currently, automated crawlers from companies like Anthropic, OpenAI, and Google are scanning extensive areas of the internet, producing outputs while sending significantly less traffic back to the original content creators than traditional search engines. This shift has left many small businesses and publishers struggling financially, despite the widespread usage of their work. Cloudflare, which protects and manages about 20% of global web traffic, and GoDaddy, the largest domain registrar in the world, have the potential to reshape the online business landscape. Together, they are advocating for new standards such as Agent Name Service and Web Bot Auth, aimed at verifying the identities of AI agents and enhancing transparency regarding their activities. The overarching objective is to establish a permission-based framework for the AI-driven internet. Instead of allowing bots to scrape content indiscriminately, this system would provide creators with clearer oversight and opportunities to benefit from the value generated by AI technologies. This initiative builds upon Cloudflare's previous efforts to limit AI crawler access by default and introduce monetization tools for content access. The implications of this partnership are significant. If AI systems continue to extract content value without reciprocating traffic, the motivation for original content creation may diminish. Cloudflare and GoDaddy are counting on this new infrastructure focused on identity, control, and compensation to help restore a viable economic model for the evolving internet landscape.