Cloudflare AI Bot Settings: What Changes on 15 September 2026

On 1 July 2026, Cloudflare rebuilt the way it grants AI bots access to a website: AI traffic is no longer a single “AI bot” box but three separate behaviours. From 15 September 2026, the default settings will also change for a subset of sites. Part of the press reported this as “every site is closing itself off to AI”; the primary source says something both narrower and more detailed. Below we cover exactly what changes, who is affected, and the three decisions a business owner needs to make.

This post is part of the AI Visibility Guide; see the guide for all four layers.

AI traffic is now three categories

In Cloudflare’s announcement dated 1 July 2026, automated AI traffic is split into three behaviours:

• Search: Behaviour that collects and indexes your content so it can later answer questions about it. In Cloudflare’s own words, “any behavior that collects or indexes your content, so it can answer questions about it later”.
• Agent: Automated behaviour acting on a person’s behalf, usually in real time, to get something done right now. An assistant that reads your product page to compare prices, or fills in a form for you, falls into this class. Software of this kind is called an AI agent.
• Training: A crawler taking your content to train or fine-tune a model.

The practical value of the split is this: “block AI bots” used to be a single button that closed all three at once. Now each can be turned on and off separately. These granular controls were opened to every customer on the Cloudflare network, free plan included.

What exactly changes on 15 September 2026?

This is the part most often misreported, so let us spell out the scope. Cloudflare’s press release published the same day lists three groups the new defaults apply to:

1. New customers,
2. New sites added by existing customers,
3. Existing free plan customers that have not changed their settings by 15 September 2026.

For these groups, Training and Agent will be blocked by default only on pages that display ads; Search stays allowed. Cloudflare’s reasoning is explicit: an ad is a signal that the page was meant for a person to land on and see. Search, on the other hand, is the behaviour that funnels visitors back, so it is left open by default.

So “every site will be blocked” is not true. Existing sites on paid plans with settings already defined are outside this automatic default change. But the opposite is not true either: saying “existing sites are not affected at all” would also be wrong. The next section explains why.

Mixed crawlers will be judged by the most restrictive rule

A second change takes effect on 15 September, and it can concern you regardless of your plan. Crawlers serving more than one purpose — specifically those combining Search with Training — will from now on be judged by all of their behaviours. Since defaults are enforced by the most restrictive applicable rule, on an account that has chosen to block Training, mixed crawlers such as Googlebot, Applebot and Bingbot will end up blocked too.

That is a real risk for sites that once switched on the one-click “Block AI bots” option. According to Cloudflare’s traffic report of 1 July 2026, more than 36% of crawler requests fall into this mixed group — not a small exception, but a large slice of traffic. Site owners who want to keep their current behaviour can opt out by marking, in their security settings before 15 September, that they want no changes to Training crawlers that also crawl for Search purposes.

In short, the danger is not “closing yourself off to AI”, as many assume; it is switching off classic search engine crawling without realising it.

Three questions a business owner has to answer

Three categories mean three separate commercial decisions. They need answering before the matter is handed to a technical team:

1. Should AI assistants recommend you to customers? If you want to appear in ChatGPT, Claude or Google’s AI answers, Search must stay open. Close it and you drop out of the source pool for those answers.

2. Should agents be able to read your product pages? Software that researches on a user’s behalf, compares prices or attempts a booking sits in the Agent class. For e-commerce and B2B sites, this category increasingly means “shop window”. We covered the commercial side of this in our earlier piece on preparing to sell to AI agents.

3. Should your content feed model training? If you produce original research, technical documentation or content with copyright value, closing Training is reasonable. The answers to these three questions do not have to match; separating the three is the whole point of the system.

Where to look in the dashboard, and what to write in robots.txt

The settings live under the relevant domain’s Security settings in the Cloudflare dashboard, and in the Bot Management configuration card. Search, Agent and Training can be switched on and off individually there, free plan included.

The finer setting that states how content may be used (use) is available to Bot Management customers and takes three values: immediate (interact, but store and reuse nothing), reference (the default: index, excerpt and link back) and full (summarise and reproduce).

You can also declare the same preference in your robots.txt file with a Content Signals line:

User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

This example means “crawlable for search, not crawlable for model training, usage at reference level”. One important distinction: the robots.txt line is a statement of preference, not a technical block. Actual blocking happens through the setting on the Cloudflare side; using both together gives the most consistent result.

Pay Per Crawl is now Pay Per Use

There is a change on the revenue side of the announcement too. The Pay Per Crawl experiment Cloudflare launched a year ago is evolving into Pay Per Use. In the wording of the press release, publishers are now paid when their content actually creates value, not just when it is fetched.

The first two implementations work like this: with Ceramic.ai, publishers are paid every time their content appears in Ceramic’s AI search results. On the You.com side, an agent can pay on demand for a specific piece of premium content it needs. We covered the wallet layer that lets agents pay on their own in our earlier piece on Cloudflare Wallets.

What to do this week

As of August 2026, there is less than a month until 15 September. A short checklist:

• Note the current state of the Search, Agent and Training settings for every domain in your Cloudflare dashboard.
• If “Block AI bots” was ever switched on, account for Googlebot and Bingbot falling into scope after 15 September; if you do not want that, mark the exemption in your security settings.
• Review free plan domains and newly added domains separately — the default change hits them first.
• If you have pages displaying ads, weigh what closing Agent access on those pages means for sales.
• Put your decision in writing with a Content Signals line in robots.txt as well.

If you want to move forward while measuring how these settings affect your search visibility, take a look at our technical SEO service; if you would like us to assess your own domains, write to us via our contact page.