Cloudflare accuses Aravind Srinivas-led Perplexity of covertly scraping data

Perplexity, an AI startup founded by Aravind Srinivas, faces serious allegations of secretly scraping data from websites that have expressly prohibited such actions. On Monday, Cloudflare, a prominent internet infrastructure company, released findings in a research blog indicating that Perplexity has been employing misleading tactics to conceal its scraping activities. According to the report, Perplexity initially presents itself with a legitimate user agent. However, when it encounters restrictions from a website, the AI startup reportedly alters its identity to bypass these blocks. This behavior raises significant concerns, as AI tools like those provided by Perplexity often rely on extensive data scraping from the internet. Cloudflare became aware of the situation after receiving complaints from clients who had implemented restrictions through their robots.txt files to block Perplexity's known bots. Despite these measures, the startup continued to access their content. Upon verifying that Perplexity's crawlers were indeed being obstructed, Cloudflare conducted tests that confirmed the company's unauthorized activities. The report noted that such activity was detected across tens of thousands of domains and involved millions of requests daily. Cloudflare was able to identify the crawler using advanced machine learning techniques and network signals. In response to these allegations, Perplexity took to X (formerly known as Twitter) to dispute the claims. The company suggested that Cloudflare's leadership might be misinformed about the fundamentals of AI. They elaborated in another post that their scraping methodology is distinct from traditional web crawling, which indiscriminately collects vast amounts of data regardless of user consent. Instead, Perplexity asserts that its user-driven agents only retrieve information upon specific requests from users, utilizing the data immediately without storing it or using it for training. Perplexity emphasized that its agents are designed to operate on behalf of users, and called for a better understanding from infrastructure providers like Cloudflare to maintain an open and accessible internet.