The short answer

Does llms.txt affect Google Search? No. Google’s AI features documentation says, in full: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” Google’s Search Central changelog entry of June 15, 2026 records the clarification directly: “Added a note to the AI optimization guide clarifying Google Search’s usage of llms.txt files.” Gary Illyes said the same at Search Central Live in July 2025. This question is closed.

Should you still ship an llms.txt? For most content sites, no, at least not yet. For documentation sites, developer tools, and API products, yes. The file has almost nothing to do with search rankings and almost everything to do with agent routing.

Why does the industry still recommend it? Because the practitioners recommending it are answering a different question than the one Google answered. That gap is the entire story, and both sides are technically correct.

What Google actually said, in order

The most common mistake in this debate is treating Google’s position as recent, or as a reversal. It is neither. The position has been consistent for roughly two years. Only the tone has softened.

Date Source Position
Sept 2024 Jeremy Howard, Answer.AI Proposes llms.txt as a community spec at llmstxt.org
Mid 2025 John Mueller, Google Compares llms.txt to the meta keywords tag: a self-declared claim rather than an observable signal
July 2025 Gary Illyes, Search Central Live Google does not support llms.txt and has no plans to
Dec 3, 2025 Google Developer Docs An llms.txt briefly appears on Google properties, then is removed
Jan 2026 John Mueller, Bluesky Confirms the file’s presence on Google properties is not an endorsement
June 15, 2026 Google Search Central changelog “Added a note to the AI optimization guide clarifying Google Search’s usage of llms.txt files”
June 2026 John Mueller, Reddit Calls llms.txt speculative, and points to WebMCP as the more interesting direction

Read the June note carefully, because it is more interesting than the headlines about it. Google does not tell you to remove the file. Its guidance is that these files are not needed for Google Search, while maintaining one is acceptable if you want to support other services or systems that use it.

That is Google itself drawing the distinction this article is about. Search does not read it. Google is not claiming nothing does.

Mueller’s meta keywords analogy is the load-bearing argument, and it is worth understanding rather than dismissing. The keywords meta tag failed because it let site owners assert what a page was about instead of demonstrating it. Search engines already had the page. Reading the owner’s claim about the page added no information and created an obvious manipulation surface. llms.txt has the same structural shape: a curated, owner-authored description of a site that a crawler could simply verify by fetching the site.

Whether the analogy holds is contested. Carolyn Shelby wrote a direct rebuttal in Search Engine Land, No, llms.txt is not the “new meta keywords”, arguing that the comparison undersells the file’s utility and that SEOs ignored meta keywords for reasons that do not transfer. Her objection is worth reading in full rather than through anyone’s summary of it, including this one.

But Mueller’s argument is not lazy either, and it explains why Google’s position has never wobbled.

The Chrome contradiction

One legitimate source of confusion: Chrome’s Lighthouse audit documentation includes a check for llms.txt while Search Central says you do not need it. Mueller addressed this directly. Different teams inside Google explore different things, and a Lighthouse check for agent readability is not a Search ranking signal. The two documents are answering different questions, which, as we will see, is the pattern for this entire topic.

What the server logs actually show

Opinions about llms.txt are abundant. Log data is not. Several independent datasets published between November 2025 and August 2026 tell a consistent story.

Ahrefs: 137,210 domains, May 2026 traffic

The single most useful study, because it measures requests rather than adoption. Ahrefs analysed server log data across every domain in its Web Analytics dataset that received traffic in May 2026.

  • 137,210 domains analysed. 28%, or 38,360 domains, published a valid llms.txt
  • 97% of those files received zero traffic in May 2026. In Ahrefs’ words: “Nothing fetched them at all”
  • Of the requests that did occur, only 19.5% came from named AI tools

The full requester breakdown is where this study stops being a debunk and starts being an argument. Almost every article citing it quotes the first line and stops.

Requester category Share of requests
SEO audit tools 21.7%
Other or unidentified 14.9%
General web crawlers 13.1%
Tech profiling tools 11.6%
AI agents and agentic infrastructure 10.5%
GEO and AEO tools 5.8%
AI training crawlers 5.3%
llms.txt discoverability bots 3.6%
Service and social bots 2.9%
Research bots 2.7%
AI assistants 2.5%
AI retrieval bots 1.1%

Two rows matter and they are nine places apart.

Agents fetch this file roughly ten times more often than retrieval bots do. In the named-bot data the pattern repeats: GPTBot leads at 4.51%, and Claude Code outfetched every AI retrieval bot and every assistant in the dataset. ClaudeBot, the training crawler, managed 0.8%. OAI-SearchBot and PerplexityBot together account for a couple of hundred fetches across 137,000 domains.

The two bots at the top of this file’s traffic are both coding agents. That is not a footnote. It is the finding.

And 21.7% deserves its own pause. The most active consumer of llms.txt files is the tooling ecosystem that audits llms.txt files. An entire measurement industry formed around a file before anyone established that the intended readers were reading it.

SE Ranking: roughly 300,000 domains

Around 10% adoption across the sample. When the researchers modelled the relationship between having the file and being cited by AI systems, using both a conventional statistical model and an XGBoost classifier, they found no measurable effect on citation frequency.

ProGEO.ai: Fortune 500, March 2026

37 of 500 companies, 7.4%, had shipped an llms.txt. For comparison, 92.8% had a robots.txt. Robots.txt was proposed in 1994 and did not reach RFC status until 2022. Standards take time, so that gap is not evidence of failure, but it is evidence of where we are.

One caveat on adoption numbers

Adoption figures vary wildly depending on what you sample, and both sides of this argument cherry-pick accordingly. A curated panel of 219 technically sophisticated hosts measured on August 3, 2026 showed 51.8% adoption. The Tranco top 1,000 showed 8.7% in June 2026.

Adoption is cohort shaped, not a single rate. Any article quoting one figure without naming the sample is doing rhetoric, not measurement.

The category error at the centre of this argument

Here is the resolution, and it is the reason two groups of competent people keep talking past each other.

Google answered the question “does this file influence Google Search?” The answer is no, with unusual clarity and consistency.

The industry is answering a different question: “should a brand publish a machine-readable surface for AI agents?” That question has nothing to do with Google Search.

These get collapsed into one debate because both live under the SEO label. They should not be. The web is splitting into two consumption layers with different mechanics:

Retrieval layer Agent layer
Who consumes it Search crawlers, RAG retrievers Coding agents, MCP clients, task agents
What it wants Your actual pages, verified A cheap, structured route into your content
Optimisation goal Be findable and citable Be usable and callable
Does llms.txt matter No, per every dataset Yes, in specific verticals
Real levers Content quality, entity clarity, third-party mentions, crawler access llms.txt, llms-full.txt, MCP servers, clean API docs

llms.txt is an agent layer artifact being evaluated with retrieval layer KPIs. Measured against AI citation share it fails, correctly, because that was never a coherent thing to measure it against. Measured against whether a coding agent can navigate your documentation without burning 40,000 tokens on your navigation menu, it works. The companies that depend on that outcome have quietly shipped it: Stripe, Cloudflare, Vercel, Anthropic, Coinbase, Pinecone, Cursor.

Notice what those companies have in common. They are all products whose users interact with them through an AI agent. That is the entire selection criterion.

And this is why the Ahrefs table above is worth reading past the headline. The study is cited everywhere as proof that llms.txt is dead. Inside it, agents and agentic infrastructure outfetch retrieval bots roughly ten to one, and the single most active named consumer after GPTBot is a coding agent. The most-cited debunk of this file contains the clearest available evidence of who does read it.

Both facts are in the same dataset. Which one you quote depends on which question you were asking.

Should you ship one? A decision table

Stop asking whether llms.txt works. Ask whether your audience uses tools that read it.

Your site Ship llms.txt? Reasoning
Developer tools, API products, SaaS docs Yes Cursor, Copilot, Claude Code and similar agents fetch docs at request time. Direct, measurable benefit today.
Technical documentation of any kind Yes Same mechanism. This is the file’s actual native habitat.
B2B SaaS with a technical buyer Probably Your evaluators use AI assistants during research. Low cost, plausible upside.
E-commerce Low priority No evidence of benefit. Product schema, feed quality and review signals matter far more.
Local business No Google Business Profile, structured data and review velocity are the levers.
Publisher or media No Your leverage is licensing negotiations and crawler policy, not a routing file.

For everyone in the no and low priority rows: the cost of shipping one is genuinely near zero, so this is not a warning. It is a prioritisation statement. If llms.txt sits on your roadmap above the items in the next section, your roadmap is wrong.

What actually moves AI visibility, in priority order

This is where the attention should go, and it is where I spend mine.

1. Do not block the crawlers you want. This sounds trivial. It is the single most common failure I find on enterprise sites. Maintaining an inventory of AI crawlers and auditing robots.txt against it, market by market, surfaces real misconfigurations. On multi-market platforms a country domain silently blocking a retrieval bot is a visibility hole that no amount of content work will close. Mueller made this same point: the most basic form of agent optimisation is not being blocked.

2. Separate training crawlers from retrieval crawlers in policy. These are different bots with different consequences. Blocking a training crawler keeps your content out of future model training. Blocking a retrieval crawler removes you from live answers. Most robots.txt files I audit treat them identically, which is almost always a mistake in one direction or the other. The distinction is documented by every major operator, and I keep a public reference of which agent does which.

3. Write extractable answers. Language models cite content they can lift cleanly: a direct claim, followed by support, in a stable structure. Marketing copy that circles a point for three paragraphs before making it does not survive extraction. This is the highest leverage content change available. If you want a number for it, produce your own rather than borrowing one: restructure a defined set of pages, hold everything else constant, and re-run a fixed prompt set before and after. Anyone quoting you a multiplier for this is selling something.

4. Build entity clarity. The model needs to know what you are, not just what you sell. Consistent naming, Organization schema with accurate sameAs, a coherent About page, and presence in the knowledge sources models actually trained on.

5. Earn third-party mentions. Independent corroboration, in forums, industry publications and comparison content you do not control, carries weight that self-published claims cannot. This is the mechanism the meta keywords analogy is really pointing at: systems trust what others say about you over what you say about yourself.

6. Then, if relevant, ship llms.txt. It belongs on the list. It belongs sixth.

How to ship one properly, in 20 minutes

If your row in the decision table said yes:

Serve it at the root. https://yourdomain.com/llms.txt, plain text, text/plain or text/markdown.

Follow the spec structure. H1 with your site name, a blockquote summary, then linked sections with a one-line description per link. The description is the part that does the work. It is what lets an agent choose a page without fetching five of them.

Curate ruthlessly. This is not a sitemap. A 400-link llms.txt is a failed llms.txt. Include what you would want an agent to cite, and nothing else.

Consider llms-full.txt separately. The companion file inlines full content as one Markdown document. Useful for documentation-heavy products, unnecessary for most sites. Do not ship it by reflex.

Do not generate per-page Markdown mirrors. A popular implementation pattern that creates a maintenance burden and a duplicate content surface with no demonstrated benefit.

Test it with a real agent. Open Claude Code or Cursor, point it at your llms.txt, and ask a question that should be answerable from the file. If the agent answers correctly without follow-up fetches, your descriptions work. If it fetches the wrong page, they are too vague. This is the only test that matters and it takes four minutes.

Instrument it. Filter your CDN logs for hits to /llms.txt by user agent. If you are going to maintain a file, maintain the evidence for whether it is being read. Most people arguing about this file have never checked their own logs.

Keep it current. A stale llms.txt is worse than none. It routes agents confidently toward pages that no longer exist.

What to watch instead

WebMCP is the more interesting development, and Mueller pointed at it deliberately. It applies the Model Context Protocol to the web, letting agents discover and invoke site functionality rather than only read it: compare products, populate a cart, submit a form. Chrome supports it. A file that describes your content and a protocol that exposes your functionality are not competitors. The second is simply a larger idea.

Standardisation status. llms.txt has no IETF RFC and no W3C working group. It remains a community convention maintained through llmstxt.org. That is not disqualifying, since robots.txt spent 28 years in the same position, but it is the honest description, and anyone selling it as a standard is overstating it.

Vendor commitments. No major model provider has published a production commitment to honour the file at crawl time. If that changes, this article changes. Watch for statements from providers, not from tool vendors whose products score you on it.

FAQ

Does llms.txt help Google rankings?
No. Google’s AI features documentation states: “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” A Search Central changelog entry dated June 15, 2026 records the clarification being added. Gary Illyes confirmed the same position at Search Central Live in July 2025. It neither helps nor harms.

Does llms.txt help with AI Overviews or AI Mode?
No. These are Google Search surfaces and follow the same policy. Google’s guidance lists the file among things you can skip.

Do ChatGPT, Claude, or Perplexity read llms.txt?
Rarely, based on log data. Ahrefs found AI retrieval bots accounted for 1.1% of requests to these files across 137,210 domains in May 2026. No provider has published a commitment to honour it.

Who actually reads llms.txt?
Primarily AI coding agents such as Cursor, GitHub Copilot, Claude Code, Continue and Cline, fetching documentation at request time, plus some MCP integrations. Secondarily, and ironically, SEO audit tools, which were the largest single requester category in the Ahrefs data at 21.7%.

Can llms.txt hurt my site?
Not directly. The realistic risks are opportunity cost and staleness: time spent on it instead of on crawler access and content structure, and an outdated file that misroutes agents.

Is llms.txt an official standard?
No. It is a community proposal published at llmstxt.org, introduced by Jeremy Howard of Answer.AI in September 2024. There is no RFC and no W3C working group.

Should I remove my existing llms.txt?
No. It costs nothing to leave in place, and removing it has no demonstrated benefit. Keep it accurate or delete it. A stale file is the only genuinely bad option.

The takeaway

Google is right. llms.txt does nothing for Google Search, and the evidence supporting that is unusually strong for an SEO claim.

The practitioners recommending it are also right, but only for the subset of sites whose users reach them through an agent, and only for reasons that have nothing to do with search.

The mistake is not picking the wrong side. The mistake is thinking there is one question here. Read the guidance, check your own server logs, and decide from your row in the table above rather than from anyone’s headline, including this one.

Emad Sharaki is a Senior SEO & GEO Strategist with 14+ years in search. He leads AI Visibility work at AUTODOC SE, tracking brand representation across six major language models and 35+ European markets, and maintains an enterprise database of 38+ verified AI crawlers used to govern crawler policy across seven European markets. He has spoken at WordCamp Porto and WordPress Day for E-Commerce, and has been quoted in Search Engine Journal.

Sources

Every figure in this article was taken from the primary source, not from coverage of it. Accessed August 14, 2026.

If you find an error in any number above, tell me and I will correct it and say so. An article arguing for primary sources should be held to that standard.