Cloudflare, a major internet Content Delivery Network (CDN), has started blocking AI crawlers from accessing content on websites without permission or compensation. The company states that this change addresses the issue of AI bots slowing down websites by crawling them more aggressively than traditional search engine crawlers. Under the new policy, AI crawlers will be automatically blocked from accessing content on any new website that signs up for Cloudflare’s services.
Site owners will have to grant explicit permission for the crawlers to access their content. Cloudflare also states that it will utilize behavioral analysis and machine learning to identify “shadow” scrapers that attempt to evade detection. This move reverses the previous status quo, where website owners had to opt out of AI crawling.
Now, blocking is the default, and AI vendors must request access and clarify their intentions. Many publishing companies, such as The Associated Press and Condé Nast, have expressed frustration over AI companies using their content without compensation or consent. Some have even ignored standard protocols that block crawlers.
Combating AI scrapers with Cloudflare
Recent court cases have ruled that the use of copyrighted works by companies such as Meta and Anthropic is legal under the doctrine of fair use. However, writers, artists, and publishers worry that the government might allow AI free rein over their content.
AI powerhouses such as OpenAI and Google are lobbying the government to classify AI training on copyrighted data as a form of fair use. The Copyright Office recently released guidelines that struck a middle ground. It stated that while some applications of generative AI could be considered a “transformative” use, the mass scraping of all data did not qualify as fair use.
Cloudflare CEO Matthew Prince said the new policy is meant to “give publishers the control they deserve and build a new economic model that works for everyone—creators, consumers, tomorrow’s AI founders, and the future of the web itself.
Cloudflare has also launched a new system that allows publishers to set their own rates for AI companies that want to scrape their content. This system, currently in private beta, will allow AI firms to pay for access or be denied if they refuse. With Cloudflare powering a considerable portion of the internet, a substantial amount of web content could become inaccessible to AI companies unless they negotiate access or pay licensing fees.
This shift could rapidly reshape the economic landscape for web content and AI.
A seasoned technology executive with a proven record of developing and executing innovative strategies to scale high-growth SaaS platforms and enterprise solutions. As a hands-on CTO and systems architect, he combines technical excellence with visionary leadership to drive organizational success.
