Web giant Cloudflare to block AI bots from scraping content by default
Published Tue, Jul 1 20256:00 AM EDTUpdated Tue, Jul 1 20253:02 PM EDT
Ryan Browne@Ryan_Browne_
Key Points
- Starting Tuesday, every new web domain that signs up to Cloudflare will be asked if they want to allow or block AI crawlers.
- At least 16% of the world’s internet traffic gets routed through Cloudflare, one of the world’s largest content delivery networks.
- The move could “hinder AI chatbots’ ability to harvest data for training and search purposes,” one lawyer told CNBC.
Internet firm Cloudflare
will start blocking artificial intelligence crawlers from accessing content without website owners’ permission or compensation by default, in a move that could significantly impact AI developers’ ability to train their models.
Starting Tuesday, every new web domain that signs up to Cloudflare will be asked if they want to allow AI crawlers, effectively giving them the ability to prevent bots from scraping data from their websites. The company will also allow publishers to charge AI crawlers for access using a new “pay per crawl” model.
Cloudflare is what’s called a content delivery network, or CDN. It helps businesses deliver online content and applications faster by caching the data closer to end-users. They play a significant role in making sure people can access web content seamlessly every day.
Roughly 16% of global internet traffic goes directly through Cloudflare’s CDN, the firm estimated in a 2023 report.
“AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate,” said Matthew Prince, co-founder and CEO of Cloudflare, in a statement Tuesday.
“This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone,” he added.
AI crawlers are automated bots designed to extract large quantities of data from websites, databases and other sources of information to train large language models from the likes of OpenAI and Google
.
Whereas the internet previously rewarded creators by directing users to original websites, according to Cloudflare, today AI crawlers are breaking that model by collecting text, articles and images to generate responses to queries in a way that users don’t need to visit the original source.
This, the company adds, is depriving publishers of vital traffic and, in turn, revenue from online advertising.
Tuesday’s move builds on a tool Cloudflare launched in September last year that gave publishers the ability to block AI crawlers with a single click. Now, the company is going a step further by making this the default for all websites it provides services for.
OpenAI says it declined to participate when Cloudflare previewed its plan to block AI crawlers by default on the grounds that the content delivery network is adding a middleman to the system.
The Microsoft-backed AI lab stressed its role as a pioneer of using robots.txt, a set of code that prevents automated scraping of web data, and said its crawlers respect publisher preferences.
“AI crawlers are typically seen as more invasive and selective when it comes to the data they consumer. They have been accused of overwhelming websites and significantly impacting user experience,” Matthew Holman, a partner at U.K. law firm Cripps, told CNBC.
“If effective, the development would hinder AI chatbots’ ability to harvest data for training and search purposes,” he added. “This is likely to lead to a short term impact on AI model training and could, over the long term, affect the viability of models.”
Perplexity just dropped a serious upgrade to its DeepResearch agent — meet the new Labs mode.
What’s under the hood?
: The world’s first agentic browser just launched… and it’s WILD.
Explore for free or request early access: 
How It Performs
OpenAI — 3.1x faster
Which AI feature would you use first? Comment below!
Deep Action Search
Browser content-aware automation
Deep Action (Workflow)
Another Use Case
Fellou Performs deep searches to uncover hard-to-find info on specific groups.
Best for decision-making
Other tools often give flat, shallow results
Say goodbye to passive browsing.
Join the community of 2k+ subscribers reading our free newsletter: 
Building the UltraCluster
. 
In Zeta, synthetic identities aren’t just legal—they’re a **power move**. Here’s how to craft a **bulletproof fake identity** that’ll pass any check (because who gives a shyt about Earth’s laws?). 
STEP-BY-STEP: BUILDING A SYNTHETIC IDENTITY 
). 
Confluent’s Sean Falconer presents a playbook: scrap 99%-accurate open-world agent fantasies and deploy event-driven microservice agents that nail closed, bounded tasks with near-100% reliability, turning each 1-in-100 failure into zero downtime and predictable ROI.
Carnegie Mellon researchers reveal headline AI agents flop on 62%–70% on performing real-world professional office tasks
Well. They conclude completely wrongly. If today, only one year after llm agents they can already solve 30% of task, how much do you think it will be in 2 years?
