Cloudflare's September 15 AI crawler change, explained for website owners
On September 15, 2026 Cloudflare started sorting AI crawlers into three groups and blocking some of them by default. About one in five websites runs through Cloudflare, so the change touches a lot of small business sites at once. Here is what changed, and the four decisions it asks you to make.
What changed
Cloudflare now treats AI crawlers as three different jobs, and lets each website answer each one separately:
| Category | What the crawler does | Examples | Cloudflare default since Sept 15 |
|---|---|---|---|
| Search | Builds search results and shows links to you | Googlebot, Bingbot, OAI-SearchBot, PerplexityBot | Allowed |
| Agent | Opens a page because a person asked an AI assistant to | ChatGPT-User, Claude-User, Perplexity-User | Allowed, except blocked on pages that show ads |
| Training | Collects content to train AI models | GPTBot, ClaudeBot, Google-Extended, CCBot | Blocked on ad-supported sites, allowed elsewhere |
The new defaults apply to new Cloudflare customers, new sites added by existing customers, and every site on the free plan. Existing paid customers keep their current settings unless they change them.
Three other pieces came with it, according to Cloudflare's announcement:
- Mixed-use crawlers are blocked by default on pages with ads. A mixed-use crawler collects for search and training at once. AI companies were given until September 15 to split them.
- Disallow AI Training is a one-click setting that refuses training while keeping the site fully in search.
- Accountable is a label for AI companies that honour robots.txt opt-outs, let sites opt out of AI summaries, show which pages they fetched, and confirm that opting out of training does not affect ranking. Apple, Google and Microsoft were named first.
Cloudflare also said its Pay Per Crawl marketplace is becoming Pay Per Use, charging AI companies when content creates value rather than each time it is fetched. That is aimed at publishers and is not something a small business needs to act on.
The four decisions
Whether or not you use Cloudflare, the change makes one thing clear: every website is now expected to say what AI may do with it. That comes down to four questions.
- Can search engines index us? Almost always yes. This includes ChatGPT search and Perplexity, which send people to your site.
- Can AI companies train on our content? Your call. Saying no does not affect search. Most businesses with original content, pricing or expertise choose no.
- Can AI agents visit on a person's behalf? Usually yes. When a customer asks an assistant "does this clinic take my insurance?", an agent fetches your page to answer. Blocking it means the assistant cannot help them.
- What must people and agents be told? If you use AI on your own site, such as a chatbot, several states already require a notice. That is what BotNotice does today.
The problem is that each answer lives in a different place: robots.txt rules, a Content-Signal line, a Content-Usage line, Cloudflare's dashboard, a licence file and a page humans can read. They drift apart. The first step is finding out what your site says right now.
The signals a website can publish
| Signal | Where it lives | What it says |
|---|---|---|
| robots.txt rules | /robots.txt | Per-crawler allow and disallow rules. The oldest signal, honoured by all responsible crawlers. |
| Content Signals | A Content-Signal: line in robots.txt | One line for three uses: search=yes, ai-input=yes, ai-train=no. Cloudflare adds it to managed robots.txt files. |
| Content-Usage | A robots.txt line or an HTTP header | The IETF's draft standard, for example Content-Usage: train-ai=n. Still a draft, but designed to become the common language. |
| RSL licence | A License: line in robots.txt | Machine-readable licence terms, including paid ones, for AI use of your content. |
| llms.txt | /llms.txt | A plain-text guide to your site for AI assistants, so they describe and cite you accurately. |
None of these is enforcement. They are requests. Enforcement happens at your host or a service such as Cloudflare. The value of publishing them is that responsible crawlers follow them, and that you have a dated record of what you asked for.
Common questions
Does this affect my site if I don't use Cloudflare?
Not directly. The defaults only change on Cloudflare. But the categories, Content Signals and robots.txt rules work anywhere, and the AI companies that agreed to separate their crawlers did so for the whole web.
Will I disappear from Google or ChatGPT?
Not from search. Cloudflare's defaults keep search crawlers on, and Google's training crawler is separate from its search crawler. What can be blocked by default is AI training and, on pages with ads, AI agents.
What is a mixed-use crawler?
One crawler that collects content for search and for AI training at the same time. Cloudflare says these were the single largest category of verified crawler traffic on its network. Since September 15 they are blocked by default on ad-supported pages unless the operator separates them.
What does 'Accountable' mean?
Cloudflare's label for AI companies that let sites opt out through robots.txt, allow opting out of AI summaries, show which URLs they fetched, and confirm publicly that opting out of training does not hurt search ranking. Apple, Google and Microsoft were named at launch.
What should a small business do?
Decide four things: search on, training on or off, agents on or off, and what people and agents are told. Then make sure your site actually says it. Our free check shows what it says today.
Sources
- Cloudflare press release, September 15, 2026
- Cloudflare: Content Signals Policy and contentsignals.org
- IETF AI Preferences working group
- Really Simple Licensing
- TechCrunch, July 1, 2026
Not legal advice. Cloudflare's settings and the standards above change; check the linked sources for the current wording.