Cloudflare’s September 15 AI-crawler defaults changed the practical meaning of “allow search, block training.” New rules separate automated traffic into Search, Agent and Training behaviors; on pages displaying ads, the stated default is to allow Search while blocking Agent and Training crawlers. For GEO work, the consequential risk is not simply losing access to a training crawler. It is misreading which crawler behavior supports AI discovery, then blocking a mixed-use bot under a more restrictive classification.
The implementation details are not settled cleanly across the available coverage. shattered.io describes the defaults as applying to domains onboarding after September 15, while hosting.com says they apply to new domains, new sites added to an existing account and free-tier sites. That difference matters enough that teams should verify their own zone status and effective policy in Cloudflare rather than infer coverage from a summary.
What changed: access is now based on behavior
Cloudflare announced its updated AI traffic options on July 1, 2026, framing the change around what a bot does with a site’s content rather than whether the operator calls it AI. Its three user-manageable use cases are Search, Agent and Training. Cloudflare Blog
- Search is proactive collection or indexing of content for answering later queries. Cloudflare says site owners should expect referral traffic or another equitable form of compensation from this behavior. Cloudflare Blog
- Agent behavior acts in real time on a person’s behalf, including chat fetch bots and browser-use agents completing a task. Cloudflare Blog
- Training behavior takes content to train or fine-tune a model, permanently absorbing that data into the model architecture. Cloudflare Blog
The September default described by the supplied reports is narrower than “Cloudflare blocks AI bots.” On ad-supported pages, Search remains allowed, while Training and Agent are blocked. shattered.io This distinction is central for GEO practitioners: a visibility policy must separate systems gathering content for retrieval and citation from systems accessing a page during a live agent task or collecting it for training.
That is a better model than a single AI-bot toggle, but it is not an automatic answer to an organization’s discoverability strategy. Search access can be useful only if the relevant crawler is actually classified as Search for the behavior being tested. The classification layer, not the name of the bot, is now the control surface.
The mixed-use problem is where GEO assumptions fail
Cloudflare explicitly says that many crawlers fit one of the three classifications, but that some fit multiple classifications. Cloudflare Blog This is not a marginal implementation detail. It is the reason a rule that looks selective in the dashboard can have a broader visibility consequence than its label suggests.
hosting.com gives Googlebot, Applebot and Bing’s crawler as examples of crawlers serving more than one purpose. It further states that Cloudflare can apply the strictest relevant rule to a crawler that spans Search and Training, meaning such a crawler could be blocked under the Training rule even if Search remains allowed. The supplied Cloudflare extract confirms the existence of multi-classification crawlers, but does not itself specify the strictest-rule behavior or list named crawler classifications. That means the exact handling of any named bot should be tested in the relevant account; it should not be assumed from category names alone.
For GEO, the operational question is therefore not “Did we leave Search on?” It is: “Can the retrieval, citation or answer-generation path we care about still fetch our pages under its actual classification?” The sources do not provide a universal crawler-to-category matrix, nor do they establish a measurement method for AI-answer visibility after a block. Evidence is thin on those outcomes. The defensible action is to validate access and downstream visibility separately.
How robots.txt fits—and where it does not replace edge controls
Cloudflare’s behavioral controls are not limited to enforcement at its network edge. According to shattered.io, its “Disallow AI Training” option includes Bot Preference Sync, which publishes the applicable no-training preference in robots.txt. That creates a second, machine-readable declaration for compliant crawlers outside of Cloudflare’s direct enforcement path.
This changes the policy architecture for teams maintaining crawler rules. A Cloudflare setting can have a corresponding robots.txt consequence, so robots directives and edge policy should be reviewed as one system. A GEO team that changes a dashboard control without checking the resulting robots.txt can miss the public policy signal it has created. Conversely, a robots.txt review alone does not establish how Cloudflare will classify or enforce traffic at the edge.
The available sources do not say that robots.txt guarantees crawler compliance. The shattered.io report characterizes Bot Preference Sync as a signal for compliant bots and describes it as additional to network-level blocking. Treat it as policy publication and corroboration, not proof that every crawler will behave as requested.
Which sites are actually affected remains a configuration question
The basic date is consistent: Cloudflare’s July 1 announcement described the new options, and the new categories were scheduled to take effect on September 15, 2026. Cloudflare Blog hosting.com But scope reporting differs.
shattered.io says existing domains were not forcibly reset because the change was framed for new domains onboarding to Cloudflare. hosting.com, however, includes new sites in existing accounts and free-tier sites among those switched to the new rules, while also saying existing manually configured sites need not change. These statements may describe different onboarding and configuration scenarios, but the extracts do not resolve that interpretation.
Do not turn this ambiguity into a generic “existing sites are safe” rule. Audit the zone, plan, site-creation timing and existing AI-bot configuration. For teams operating multiple domains, this should be a per-zone review, not a one-time account-level declaration.
A GEO test plan for the new defaults
The sources support a narrow but useful test plan. It is designed to establish access policy and catch classification overlap; it cannot, on its own, prove that a brand will appear in any particular AI answer.
- Inventory pages that display ads, because the reported September defaults turn on that page characteristic. shattered.io
- Review the dashboard’s Search, Agent and Training policies rather than relying on a legacy “AI bots” mental model. These are the three categories available for all customers to manage in Cloudflare’s stated taxonomy. Cloudflare Blog
- Identify the crawlers tied to your search, retrieval, citation and agent-use cases, then check whether any have multiple purposes. Cloudflare acknowledges that some crawlers span classifications. Cloudflare Blog
- Inspect robots.txt after enabling a no-training preference, since Bot Preference Sync may publish that choice there. shattered.io
- Re-test after policy changes. hosting.com says the settings can be changed when issues are observed; that makes policy review iterative rather than permanent.
The real decision is a visibility trade-off, not a universal block
Cloudflare’s taxonomy gives publishers more control, but it also makes crawler policy a more explicit GEO decision. Ad-supported publishers may reasonably prioritize restricting Agent and Training access on pages where direct human visits produce revenue. Sites whose economics depend more on being discovered have a different trade-off. hosting.com argues that blocking can reduce discoverability for service businesses, portfolio sites and product-supporting blogs.
That conclusion is an editorial assessment, not a quantified result in the supplied evidence. There is no traffic-loss figure here, no universal ranking effect and no documented guarantee that allowing a Search crawler produces a citation or referral. What is documented is the change in default access, the behavioral categories and the possibility of multi-purpose crawlers. That is enough to justify careful testing—and not enough to justify confident claims that an AI block is either harmless or catastrophic.