The Internet architecture provider Cloudflare its strategy has changed in relation to content extractors for AI (Scrapers). From now will block them by defaultavoiding that they can access the contents of their clients’ websites without permission to do so or without receiving any financial compensation for it. This has been confirmed by the company in a statement, which reflects that the new domain owners will begin to ask if they want their contents to be accessible to content extractors for ia.
Will also let certain online media and platforms can implement a “payment per extraction” modelwhich also opens the door to a new business model that will make the AI companies that want to track and collect their content to train their models or to use it with AI agents and other artificial intelligence systems have to compensate for the owners of the contents to be able to collect it.
This tracking payment program will allow editors to set a Price so that content extractors can access those who own and have published online. IA companies can then see prices and decide whether they are registered in the program to pay for the requested fee or reject it.
For now, the implementation of the program has only reached a group of the main editors and content creators of the world, but Cloudflare has confirmed that it will be ensured to expand it and that. IA companies can use quality content in the right way. That is, with permission and financial compensation for doing so.
Cloudflare has been helping domains owners for a while to fight these content extractors. They began by allowing the websites to block the AI trackers in 2023, but only to those who meet the conditions established in robots.txt, the file in which it is clear if the bots of any kind can access the contents of the web.
Cloudflare began allowing the websites to block all the IA bots last year, whether they met the conditions of the Robots.txt files of the websites as if they did not, and it is the option they have activated by default all new clients of Cloudflare.
Last March, Cloudflare also activated a function that sends to the web content extraction bots to a kind of “Labyrinth of AI” to try to get companies that had deployed them to collect content without permission.
Among the media, editors and online platforms that will already be able to participate in this cloudflare program are Associated Press, ATLAS OBSTURA, Buzzfeed, Condé Nast, USA Today, O’Reilly Media, Pinterest, Reddit, Sky News, Sourceforce, The Atlantic, Fortune, Stack Overflow, Time and Quora.
In addition, Cloudflare ensures that you are working with AI companies to help verify their trackers and allow them to establish and indicate what their purposes are, and what they want to use: Use the content for training, inference or search. Websiters can review this information and decide which trackers let their contents access.
Matthew Prince, CEO and one of the founders of Cloudflarehas stressed that «If the Internet will survive the AI era, we need to give editors the control they deserve, and develop a new economic model that works for all: creators, consumers, the founders of tomorrow, and the future of the web itself. The original content is what makes the Internet one of the main inventions of the last century, and it is essential that the creators continue to develop it. IA trackers have been extracted without limit. Our goal is to put power in the hands of the creators, while helping AI companies to innovate. This goes for the future protection of a vibrant and free internet without a new model that works for all«.
