Who Gets to Crawl Your Website? Cloudflare Just Changed the Answer

‍

For decades, the web operated on a straightforward bargain. Publishers made their content publicly available, and search engines indexed it, then sent traffic back in return. It was an imperfect system, but it had logic: both sides got something of value. That bargain is now under serious strain, and Cloudflare's new AI crawler controls are the clearest sign yet that the infrastructure layer of the internet is adapting to a new reality.

‍

AI training bots disrupted the original deal. They consume content at scale - sometimes billions of pages - to improve language models, without returning traffic, attribution, or revenue to the publishers whose work they processed. The content fuels the model. The model competes with the content. The publisher gets nothing. Cloudflare's response is to give publishers a formal mechanism to decide who gets access, on what terms, and for what purpose.

‍

The Old Rules No Longer Apply

‍

Search engines built their entire value proposition on reciprocity. Index a site, drive visitors to it, repeat. Publishers tolerated the crawling because the trade was visible and measurable. AI training bots operate under a completely different model - one where the publisher's content is an input, not a destination.

‍

This is not a concern limited to large media companies or professional publishers. Any business with a website that generates revenue from traffic, leads, or advertising is affected. A regional law firm, a niche e-commerce site, an independent news outlet - all of them are producing content that AI systems are free to harvest under the current default state of the web.

‍

Cloudflare's new controls represent a formal acknowledgment that not all bots carry the same intent or create the same relationship with a publisher. The company is not taking a political position on AI. It is recognizing that infrastructure should reflect reality - and the reality is that the bot ecosystem has fragmented into categories with very different implications for site owners.

‍

Three Types of Bots, Three Very Different Relationships

‍

Cloudflare now distinguishes between three categories of automated traffic: search bots, agent bots, and training bots. Understanding the difference is essential before making any decisions about your own site.

‍

Search bots - like Googlebot - remain the foundation of web discoverability. They index your content and, when someone searches for a relevant term, they send that person to your page. The original bargain holds here. Most publishers have good reason to keep welcoming these bots.

‍

Training bots harvest content to build or refine AI models. They return nothing directly to the publisher - no traffic, no link, no credit. These are the most contested category, and the one generating the most friction between AI companies and content creators across the industry.

‍

Agent bots act on behalf of users - booking appointments, comparing prices, pulling specific answers from pages. They are genuinely useful to the end user, but they may complete entire tasks without the user ever visiting your site. For ad-supported publishers, this erodes page views and impressions directly.

‍

Starting September 15, 2026, new domains on Cloudflare will block agent and training bots by default on ad-supported pages. Some bots serve dual purposes - indexing for search while also harvesting for training. Cloudflare's controls allow publishers to separate those functions and make deliberate choices about each one.

‍

Why These Settings Demand Strategic Thinking

‍

Most site owners have never audited which bots are crawling their content, let alone what those bots do with it afterward. That gap between technical reality and business awareness is exactly why default settings matter so much - and why accepting them without review carries real risk in either direction.

‍

The case for restricting training bots is straightforward for any publisher who depends on advertising or subscription revenue. Your content is being used commercially without compensation. Blocking those bots is a reasonable defensive posture. But the case for blanket blocking breaks down quickly when you consider the other side of the equation.

‍

Cutting off all AI-related crawlers could reduce your visibility in AI-powered search tools like Perplexity, or in future AI-native surfaces where recommendations and citations drive discovery. If your business benefits from being referenced or recommended by AI assistants, restricting access too aggressively may trade short-term data protection for long-term audience reach. The right answer is not the same for every business, which means these settings cannot be treated as a one-size-fits-all checkbox.

‍

The starting point is mapping your site's pages to their purpose. Ad-supported content pages are strong candidates for blocking training and agent bots - the new Cloudflare default reflects exactly this logic. Pages designed primarily for discoverability, like blog posts or guides, may benefit from a more permissive approach to search-adjacent bots while still blocking pure training access. Your robots.txt file and Cloudflare settings work together but are not identical in scope, so both need review. Treat any policy you set today as a living document - the crawler landscape is changing fast enough that a decision made now may need revisiting within six months.

‍

A Broader Shift in Who Controls the Web

‍

Cloudflare's move is one piece of a larger negotiation happening across the internet about the value of content and the terms under which AI systems can access it. Publishers, news organizations, and independent creators are pushing back against a system where their work trains models that then compete with them for the same audience.

‍

Expect more infrastructure providers and hosting platforms to offer similar controls as publisher demand grows. Regulatory pressure - particularly in the EU - may eventually formalize these distinctions, turning what is currently a technical option into something closer to a legal right. The direction of travel is clear even if the destination is not fully mapped yet.

‍

The web's founding principle was open access. But open access was never meant to be a blank authorization for commercial extraction without reciprocity. Publishers who build clear, intentional crawler policies now will be better positioned as industry norms solidify around these questions. The ones who leave their settings on default - in either direction - are simply letting someone else make the decision for them.