AI site Perplexity uses “stealth tactics” to flout no-crawl edicts, Cloudflare says

The AI search engine Perplexity is under scrutiny as Cloudflare, a prominent network security and optimization service, claims it is employing covert methods to bypass no-crawl directives set by websites. This accusation, if substantiated, contravenes established internet protocols that have been upheld for over thirty years. In a recent blog post, Cloudflare revealed that it had received multiple complaints from clients who had taken measures to prevent Perplexity’s scraping bots from accessing their content. These measures included configuring their robots.txt files and utilizing web application firewalls to block known Perplexity crawlers. However, despite these restrictions, Cloudflare observed that Perplexity continued to gain access to the sites’ data. Following these reports, Cloudflare's researchers conducted their own investigation. They discovered that when Perplexity's recognized crawlers encountered blocks from robots.txt files or firewalls, the company resorted to using stealthy bots that employed various techniques to conceal their activities. According to the researchers, this clandestine crawler utilized numerous IP addresses not registered within Perplexity’s official range, frequently rotating through these addresses to evade detection. Furthermore, Cloudflare noted that requests from different Autonomous System Numbers (ASNs) were observed, indicating an ongoing attempt to circumvent website restrictions. This evasive behavior was detected across a vast number of domains, generating millions of requests daily. If these allegations are accurate, they represent a significant breach of internet etiquette established decades ago. The Robots Exclusion Protocol, proposed by engineer Martijn Koster in 1994, created a standardized way for websites to communicate restrictions to crawlers through a simple robots.txt file. This protocol, which was formally recognized as a standard by the Internet Engineering Task Force in 2022, has been widely adopted and respected throughout the internet community.