Common Crawl’s review of 584,107 llms.txt files finds a convention beginning to split into incompatible jobs. Sites are using the file as a curated content guide, a sitemap-like URL dump, a template artifact and, in some cases, a crawler-policy document—even though llms.txt “grants nothing and blocks nothing.” Search Engine Journal
That matters for GEO work because a public statement about AI access is only useful if it matches the technical mechanism that a relevant crawler follows. Common Crawl says CCBot honors robots.txt, not llms.txt. A restriction written only in llms.txt therefore does not restrict CCBot; it creates a visible policy claim that can disagree with live access rules. Search Engine Journal
This is evidence of convention drift, not proof of a settled standard
The important finding is not simply that some publishers misunderstand a file format. It is that the same root-level filename is accumulating several meanings in production. The proposed purpose of llms.txt is to provide AI systems with a curated list of a site’s pages. In contrast, robots.txt communicates crawler access rules. Those are adjacent operational concerns, but they are not interchangeable. Search Engine Journal
For a GEO practitioner, that distinction is practical. A file that points to current documentation, policies, product pages and canonical explainers is a content-selection decision. A file that says a crawler is disallowed is an access-policy assertion. Treating both as one task makes it too easy to believe that a site has implemented a crawler restriction when its robots.txt says otherwise.
Ken Ashe describes the useful model as “context packaging”: a way to tell an AI system where to start in understanding a site, rather than a way to say it may not crawl a page. That is a sensible interpretation of the observed data, but it should not be inflated into evidence that AI systems use the file. Common Crawl explicitly says it cannot confirm whether any AI system reads llms.txt. Ken Ashe Search Engine Journal
The CCBot mismatch is the clearest operational warning
Common Crawl found 1,570 analyzed llms.txt files that named a specific crawler. Thirty-two appeared to deny CCBot. Of the 31 corresponding sites the team could assess, none blocked CCBot outright in robots.txt: five explicitly allowed it, 11 restricted some paths while allowing others, and 15 had no restrictions. One additional site returned HTTP 429 and could not be tested. Search Engine Journal
This is more than a syntactic mistake. It produces contradictory public signals. One cited example says in llms.txt that CCBot has been blocked “as of June 2026,” while the site’s robots.txt permits the crawler through a wildcard rule. Search Engine Journal
For GEO teams, the working rule is straightforward: separate a site’s desired model context from its crawler-access policy. If the goal is access control for a crawler that respects robots.txt, the control belongs in robots.txt. If the goal is to help an AI system find high-signal material, llms.txt can describe that material—but it should not pretend to impose a restriction it cannot enforce. SEO Vendor reaches the same conclusion: use llms.txt as descriptive guidance and keep enforceable directives in the appropriate controls. SEO Vendor
Template-led adoption makes file counts a poor GEO metric
The dataset also argues against treating llms.txt presence as a visibility KPI. Search Engine Journal reports that 68% of the files came from a plugin or template, with Wix accounting for 41% of the files examined. Only 49% had the full structure tested in the analysis: an H1, a summary blockquote and link sections. Meanwhile, 22.56% contained no links at all. Search Engine Journal
Those results do not show that templates are harmful. They show that existence does not reliably indicate editorial intent. A generated file may be structurally valid yet offer no selected routes into a site’s useful material. GoDaddy parked-domain files, for example, passed the structural checks despite having no links, while one All in One SEO pattern contained a median of 138 links and was characterized as treating llms.txt like a sitemap. Search Engine Journal
That leaves GEO teams with a more demanding standard than “publish the file.” A maintained file should reflect a deliberate answer to which pages best explain the organisation, its products, documentation and policies. Ashe’s advice to keep it short, link to canonical pages and remove stale documentation follows directly from that distinction between curated context and automated output. Ken Ashe
Policy language is spreading beyond the file’s technical role
Common Crawl found that 6% of files included AI site-usage guidelines such as rate limits and copyright notices, areas the llms.txt standard does not cover. It also found 136,578 robots.txt files served at the llms.txt path—about 10% of 1,287,207 HTTP 200 responses at that path. Search Engine Journal
Taken together, these are signs of a policy vocabulary without a consistently shared implementation model. Some operators are publishing descriptive content guidance; others appear to be publishing requests for crawler behavior; still others are serving robots.txt itself at the llms.txt location. That is enough to justify an audit of conflicting conventions. It is not enough to claim a uniform industry practice or a durable growth trend.
The limits are material. The sample came from sites CCBot had recently fetched successfully, not the entire web. It was based on a single crawl, so it cannot establish whether templates, policy language or prompt injections are increasing. Counts for policies and injections were based on keyword matches and are likely undercounts. Search Engine Journal
What this changes in a GEO review
I would treat llms.txt as a content-governance artifact, not an AI-access-control dashboard. Its value, if a downstream system chooses to read it, is in reducing ambiguity about the pages a site wants understood. Its presence alone says little, especially in a template-heavy ecosystem. The available evidence does not establish a direct ranking effect, broader indexing or stronger local visibility from publishing the file. SEO Vendor
- Compare every crawler-specific statement in llms.txt with robots.txt, rather than assuming the first file creates enforcement.
- Remove policy language from llms.txt when it conflicts with technical controls or cannot be honored by the file format.
- Assess the linked pages as editorial choices: current, canonical and genuinely useful pages are more meaningful than a long automated list.
- Do not report llms.txt deployment as evidence that AI systems have read, honored or cited the site.
The strongest lesson from Common Crawl is not that llms.txt has failed. It is that its practical meaning is being decided by plugins, site builders and inconsistent publisher behavior as much as by its proposal. For GEO, the response is not another checkbox. It is making content guidance, declared policy and actual crawler controls say the same thing. Search Engine Journal