HasData found that 118 sites which had Cloudflare-managed AI restrictions in archived robots.txt files no longer carried those restrictions on September 16. The affected files had previously combined an ai-train=no Content-Signal, an EU copyright reservation, and a GPTBot ban; none of the 118 had that managed block in the later check. ppc.land
For GEO work, the important change is not merely that an opt-out can disappear. It is that bot policy now has to be treated as deployed configuration: something to retrieve, inspect, compare, and test. A team cannot safely use a Cloudflare dashboard state, a CMS template, or a historic robots.txt export as evidence that its current public policy says what it intends to say.
The failure mode is a sync problem, not just a robots.txt problem
HasData’s September re-run covered the same 10,894 domains it had measured in July. Among Cloudflare sites, the share disallowing GPTBot in robots.txt fell from 17.1% to 9.9%; outside Cloudflare, the equivalent figure changed only from 18.7% to 18.6%. The comparison does not prove a controlled cause-and-effect relationship, and HasData explicitly frames it as change over time rather than a controlled experiment. But the timing matters: Cloudflare changed its controls on September 15, and a customer migration window was described as beginning the following week. ppc.land
To investigate the fall, HasData looked at archived robots.txt copies for 165 Cloudflare sites that stopped disallowing GPTBot. Archived copies were available for 133. Of those, 118 had Cloudflare’s managed block in the earlier crawl; 107 were from August and 11 from July. HasData says sites that wrote their own rules retained them, while sites relying on Cloudflare to write the rules were left with an empty file. ppc.land
That distinction is operationally useful. A policy maintained directly in a repository or CMS still needs checking, but it has a different failure surface from a policy generated by an edge platform. When a platform retires, migrates, or renames a control, the question is not simply whether the new control exists. The question is whether the public artifact and the actual request path now reflect its selected state.
Cloudflare changed both the controls and the policy model
Cloudflare announced its Disallow AI Training setting on September 15. Its stated purpose is to let sites remain discoverable in search while declining AI training by mixed-use crawlers, where one crawler can serve both purposes. Cloudflare says Apple, Google, and Microsoft honor or have committed within specified time frames to honor the setting. blog.cloudflare.com
The distinction between this setting and an outright block matters. Cloudflare classifies bot activity as Search, Training, or Agent. Under the account of the rollout published by Samir Sengupta, Disallow AI Training is a Training control: accountable mixed-use crawlers can still crawl for search, while other training crawlers are blocked. By contrast, the Block setting now applies to mixed-use crawlers too, including Applebot, Bingbot, and Googlebot, and can therefore stop search crawling as well. samcodeman.com
Cloudflare also says a robots.txt directive cannot itself identify a crawler, determine its purpose, or stop an operator that ignores the directive. Its position is that the network layer can publish a preference, identify and classify crawlers, block those that do not comply, and report operator behavior. blog.cloudflare.com
That is a sensible reason not to reduce an audit to a single file fetch. It is not, however, a reason to stop fetching the file. Public robots.txt remains the machine-readable statement that other systems and teams will see. If it is empty when the organization expects a no-training preference, there is a policy-observability failure even if an edge rule continues to refuse particular requests.
The live file and live enforcement must be separate checks
HasData’s broader results show why these checks cannot be substituted for each other. In its enforcement subset, 592 sites disallowed GPTBot in robots.txt, but 234 of them, or 39.5%, still returned a live page to its GPTBot request. HasData calls this a paper-only ban. Conversely, it found sites that blocked GPTBot despite saying nothing about it in robots.txt, which it calls silent enforcement. ppc.land
There are limitations to carry into an audit. HasData used GPTBot’s official user-agent string, but its requests were not OpenAI’s verified crawler. A server that validates identity beyond a user-agent could therefore respond differently to the test than it would to OpenAI. About 15% of enforcement requests also hit proxy errors and were excluded, while 8.2% received a challenge or CAPTCHA and were reported separately. ppc.land
Those limits do not invalidate the audit model; they define it. Test results are evidence about the exact request identity, network, and time used. They are not universal proof of how every crawler will be treated. Keep the request details alongside the result rather than translating one 200 or 403 into a broad conclusion about all AI access.
For a GEO team, a practical audit record should include:
- the current response body and status for /robots.txt, including the host and protocol actually used;
- the expected directives for the crawlers and uses that matter to the site;
- the Cloudflare control state and the date it was last reviewed;
- request-level tests for the relevant crawler identities, with a normal-browser control; and
- before-and-after snapshots whenever bot settings, CDN configuration, or managed robots features change.
This is configuration assurance, not a claim that every AI operator will honor a published preference. Cloudflare itself says robots.txt alone cannot stop a crawler that ignores it. blog.cloudflare.com
Do not overstate what Disallow AI Training currently covers
There is also a material implementation gap for Bing. Microsoft’s robots.txt no-training support is targeted for early 2027, according to the account of Cloudflare’s rollout. Until then, selecting Disallow AI Training does not automatically convey a no-training preference to Bing through robots.txt; the described current Bing route is the NOARCHIVE meta tag, alongside Bing’s URL or content-removal tools. samcodeman.com
Nor should teams assume that Bot Preference Sync is visibly populating every public file. In HasData’s September collection of 6,451 robots.txt files, Bot Preference Sync appeared in none. A next-day live recheck found an AI-training signal on one of 101 affected sites that answered: Patreon, which HasData says wrote its own scoped Content-Signal line. ppc.land
The narrow lesson is durable: dashboard intent, published policy, and observed enforcement are three different objects. GEO practitioners should record all three. When they match, the organization has an explainable bot-policy posture. When they diverge, the divergence itself is the finding—and it should be resolved before anyone reports that a site is open to, closed to, or selectively available to AI systems.