Pay-per-output? AI firms blindsided by beefed up robots.txt instructions.

Prominent internet companies and publishers, including Reddit, Yahoo, Quora, Medium, and The Daily Beast, are rallying around a new initiative aimed at curbing the unauthorized scraping of their content by AI crawlers. On Wednesday, the 'Really Simple Licensing' (RSL) standard was unveiled, introducing an automated licensing framework designed to prevent bots from exploiting content without proper permissions or compensation. This innovative RSL standard builds upon existing robots.txt instructions, adding a layer that clarifies the terms of use, licensing, and remuneration for any digital content utilized in AI training. As of now, this protocol is freely available for all publishers to adopt. Created by the RSL Collective, which includes notable figures like Doug Leeds, the former CEO of Ask.com, and Eckart Walther, a past Yahoo vice president and co-creator of the RSS standard, the RSL aims to protect various forms of digital content—ranging from websites to books and videos. The framework allows for diverse licensing models, including free access, attribution requirements, subscriptions, pay-per-crawl (where publishers receive payment each time their content is crawled), and pay-per-inference (where publishers earn compensation whenever their content is used to generate AI responses). Leeds shared with Ars that the inspiration for leveraging the RSS model to develop the RSL emerged during a presentation he hosted at the University of California, Berkeley, late last year. This discussion with Walther, an old friend, spurred thoughts on the transformative impact of AI on the search landscape, particularly as publishers face dwindling traffic due to AI-generated outputs that reference their work.