Cloudflare broke much of the Internet with a corrupted bot management file

A significant outage at Cloudflare recently affected numerous websites and online services, initially leading the company to suspect a massive DDoS (Distributed Denial-of-Service) attack. Cloudflare's co-founder and CEO, Matthew Prince, expressed concerns about the potential involvement of the notorious Aisuru botnet in internal discussions. However, further investigation revealed that the root of the problem was not an external attack but rather an internal glitch. A crucial file within the system unexpectedly doubled in size, which caused issues for the software relying on it to maintain Cloudflare's bot management system—designed to safeguard against various security threats. The outage impacted Cloudflare’s core CDN and security services, among others. In a detailed post-mortem, Prince explained that the issue stemmed from a change in permissions within one of their database systems. This change resulted in multiple entries being outputted into a single 'feature file' used by the Bot Management system, causing it to swell in size. This oversized file was then propagated across the entire network, leading to the software failures. The affected machines, responsible for routing traffic on the Cloudflare network, rely on this feature file to adapt to evolving threats. Unfortunately, the software was not equipped to handle the increased size of the file, which caused it to malfunction. Ultimately, after reverting to a previous version of the feature file, the main flow of traffic gradually returned to normal. Nevertheless, it took an additional two and a half hours to manage the heightened load on various network components as traffic surged back online.