Does llms.txt work? AI search myths checked against Google's own guide

In May 2026 Google published a guide to its AI search features that settles most of the "AI hacks" debate for Google Search. Here is each popular tactic, what Google says, what independent tests found, and what to do instead.

Last reviewed September 18, 2026.

The short answer

—llms.txt does nothing for Google Search, including AI Overviews and AI Mode. Google's guide says Search ignores it. It is harmless to keep for other systems.

—No special schema, chunking or "AI writing style" is required. The one large controlled test of adding schema found no citation uplift.

—Some tactics carry real risk. A page for every fan-out query can breach Google's scaled content abuse policy, and since May 2026 Google's spam definition covers attempts to manipulate generative AI responses.

—Three switches are often confused. Google-Extended doesn't remove you from AI Overviews. Blocking GPTBot doesn't remove you from ChatGPT search. And since September 15, 2026, Cloudflare's "Block" setting for AI training also blocks Googlebot.

—What matters is plainer: crawlable pages, clear answers to the questions about you, and mentions in the sources Google already cites.

The source for most of this is Google's guide to optimizing for generative AI features in Search, published May 15, 2026 and last updated July 10, 2026. Its position on the new acronyms is one sentence: "From Google Search's perspective, optimizing for generative AI search is optimizing for the search experience, and thus still SEO."

The ledger

Does llms.txt work?

Not for Google Search. Google's guide is explicit: "You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."

It also says keeping one is harmless: "It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them."

This was Google's informal position before it was written down. In April 2025 John Mueller wrote on Reddit, as reported by Search Engine Journal: "AFAIK none of the AI services have said they're using LLMs.TXT (and you can tell when you look at your server logs that they don't even check for it)." He compared it to the keywords meta tag.

Outside Google, the crawler documentation that OpenAI, Anthropic and Perplexity publish for their search bots doesn't say those bots read llms.txt, as of this writing.

Do this instead: put the facts you want quoted (prices, policies, specifications, claims) in the HTML text of pages that are crawlable and eligible for a snippet. That is Google's stated eligibility bar for AI Overviews and AI Mode.

Do you need schema markup for AI Overviews?

Not to be cited. Google's guide: "Structured data isn't required for generative AI search, and there's no special schema.org markup you need to add." Its AI features page (updated December 10, 2025) says the same: "There's also no special schema.org structured data that you need to add."

The best test we know of agrees. In Ahrefs' study published May 11, 2026, 1,885 pages that added JSON-LD between August 2025 and March 2026 were compared with 4,000 matched pages that didn't, 30 days before and after. AI Overview citations for the schema pages fell 4.6% relative to the controls, a small but statistically significant decline. AI Mode (+2.4%) and ChatGPT (+2.2%) moved by amounts Ahrefs called "statistically indistinguishable from zero." Its conclusion: "Adding schema produced no major uplift in citations on any platform."

One caveat: every page in that test already had 100 or more AI Overview citations. We haven't found an equivalent public test on pages that aren't cited yet.

The myth survives because cited pages often do carry schema. Well-built sites tend to have structured data and good content at the same time. That is a correlation, and the test above is the closest thing we have to measuring the cause.

Do this instead: keep structured data that matches the visible text, which is on Google's own best-practice list for AI features, and use it where it earns rich results or feeds shopping. Don't add it expecting citations.

Should you chunk content or write "for AI"?

No. Google's guide: "There's no requirement to break your content into tiny pieces for AI to better understand it." And: "There's no ideal page length, and in the end, make pages for your audience, not just for generative AI search." On style: "You don't need to write in a specific way just for generative AI search."

Danny Sullivan said it more bluntly on Google's Search Off the Record podcast in January 2026, as transcribed by Search Engine Land: "So we don't want you to do that. I was talking to some engineers about that. We don't want you to do that. We really don't."

The research behind "writing for AI" is thinner than its reputation. The 2024 academic paper that named the field, GEO: Generative Engine Optimization (Aggarwal and others, KDD 2024), found that adding quotations raised a source's visibility in generated answers by about 41% and adding statistics by about 31%, while keyword stuffing lowered it by about 8%. Those results came from a simulated answer engine, not from Google, and measure the share of an answer's text attributed to a source. A 2025 benchmark, C-SEO Bench, tested such methods and found that "most current C-SEO methods are not only largely ineffective but also frequently have a negative impact on document ranking". The same paper found traditional SEO strategies "significantly more effective".

What the 2024 paper rewarded (sources, statistics, quotations, clear prose) is what good writing for people already looks like. Do it because it makes the page more useful and more quotable, not as a trick.

Bing's advice differs in emphasis. Its February 2026 announcement of AI citation reporting says: "Clear headings, tables, and FAQ sections help surface key information and make content easier for AI systems to reference accurately." Clear structure written for readers satisfies both.

Should you create a page for every fan-out query?

No. Google's AI features use query fan-out: the model runs related searches on subtopics to gather more results before it answers. That explains why pages that don't rank for the main query still get cited. It doesn't make one page per sub-question a good idea.

Google's guide addresses it directly: "While it might be tempting to create separate content for every possible variation of how people might search (for example, by focusing on other queries that people have asked, or fan-out queries), doing so primarily to manipulate rankings or generative AI responses in Google Search violates Google's scaled content abuse spam policy."

Do this instead: answer the real sub-questions on the page that answers the main one, and make sure that page ranks. Our guide to getting cited covers how fan-out changes which pages get picked.

Can you seed mentions, or publish lists that rank yourself first?

This is the riskiest group. Google confirms that third-party mentions matter: "Just like the rest of Google Search, our generative AI features can show what's being said about products and services across the web, including in blogs, videos, and forum discussions."

It also says manufactured ones don't work: "However, seeking inauthentic "mentions" across the web isn't as helpful as it might seem. Our core ranking systems focus on high-quality content while other systems block spam; our generative AI features depend on both."

Google's spam policies now define spam to include "attempting to manipulate generative AI responses in Google Search". Search Engine Land reported the wording change on May 15, 2026.

Self-ranking "best X" lists are a softer version of the same tactic. In February 2026, SEO consultant Lily Ray documented visibility drops of 29% to 49% at SaaS and B2B sites that had published many "best X" lists ranking themselves first, in some cases hundreds of them. She quoted Glenn Gabe: "For sites dropping with the latest volatility, they are dropping in AIOs as well."

Do this instead: earn real mentions in the sources Google already cites for your category, and publish honest comparison pages. Our earned-media playbook goes through each source type.

Does updating the date make content look fresh to AI Overviews?

There's no sign it helps AI Overviews. In a July 2025 Ahrefs study of about 17 million citations across seven platforms: "all AI assistants show a clear preference for fresher content—except Google's AI Overviews, which cites content that is 16 days older, on average."

Other assistants may weigh recency more, so updating pages when the facts change is worth doing. Changing a date without changing the content is a different thing, and it costs you credibility with readers who notice.

Do this instead: update pages when prices, policies or facts change, and keep the date honest.

The wrong switches: Google-Extended, GPTBot and Cloudflare's "Block"

Google-Extended doesn't take you out of AI Overviews. Google's crawler documentation (updated July 14, 2026): "Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." It governs whether Google may use your content to train future Gemini models (and for grounding in Gemini apps and Vertex AI).

The switch for AI Overviews and AI Mode is Search Console's Search generative AI control. Google says: "As of August 31, 2026, we've rolled out this control to all websites worldwide." It covers AI Overviews, AI Mode and generative AI features in Discover, and Google says it isn't used as a ranking signal elsewhere in Search. The cost is stated plainly: "You won't receive any traffic or impressions from these features." The older levers, nosnippet and max-snippet, also limit your ordinary search snippets. We explain the control and the new report next to it in our Search Console article.

Blocking GPTBot doesn't take you out of ChatGPT search. OpenAI's crawler documentation separates the two: "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." Search is OAI-SearchBot: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." And: "Each setting is independent of the others".

Cloudflare's "Block" setting can now take you out of Google. In a September 15, 2026 post, Cloudflare wrote: "Block and 'Block on pages with ads' now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training." Its new "Disallow AI Training" option is the one that refuses training while keeping search. Of everything on this page, this is the setting most likely to hurt you by accident. If your brand vanished from Google this month, check it before you rewrite anything.

What about tools that see "internal" Google data?

Google's guide: "Be wary of third-party tools that promise ranking success or claim to use 'internal' Google metrics."

No outside tool has Google's internal AI data. What you can see is Google's own Search Console reporting and the answers themselves, captured with a date and a country. Our reporting guide shows how to combine the two.

What's different outside Google Search

• —ChatGPT: allow OAI-SearchBot if you want to appear in ChatGPT search. GPTBot is a separate, training-only decision.
• —Bing and Copilot: Bing's advice favors headings, tables and FAQ sections, and Bing Webmaster Tools now reports AI citations and the "grounding queries" behind them.
• —JavaScript: in Vercel and MERJ's December 2024 analysis, the crawlers from OpenAI, Anthropic, Meta, ByteDance, Perplexity and Common Crawl did not execute JavaScript; Google's Gemini (through Googlebot) and AppleBot did. Content that only appears after JavaScript runs may be invisible to most assistants outside Google.

What actually matters

1. 01Be eligible. Indexed, crawlable (check robots.txt and your CDN), snippet-eligible, and the Search Console AI control left on unless you chose otherwise.
2. 02Be the clearest source on questions about you. In our 50 captures of Notion's AI Overviews, Notion's own pages were cited for refunds, pricing and uptime, and almost nowhere else.
3. 03Be mentioned by the sources Google already cites for your category, without faking it.
4. 04Publish things worth citing: specific facts, original data, named sources.
5. 05Measure with dated evidence, not with a tool's promise.

What we don't know yet

• —Google hasn't published how AI Overviews choose among eligible pages beyond its description of core ranking systems plus query fan-out.
• —The schema test covered pages that were already heavily cited. The effect on pages that aren't cited yet is untested in public.
• —The only large public measurement of AI crawlers and JavaScript is from December 2024.
• —llms.txt could matter to some assistant or agent in the future. As of this writing, none of the major AI search crawlers' documentation says it reads the file.

FAQ

Does Google use llms.txt? No. Google's guide says Google Search "ignores them", and that keeping the file neither harms nor helps your visibility in Google Search.

Should I delete my llms.txt file? There's no need. Google says it's fine to keep one for other systems that use it.

Does FAQ or other schema help you get into AI Overviews? Not as far as the evidence shows. Google says no special markup is needed, and Ahrefs' May 2026 test of 1,885 pages found no citation uplift from adding schema. FAQ content can still help readers.

Is GEO different from SEO? Some call this work GEO or AEO. Google's guide calls it SEO: "optimizing for generative AI search is optimizing for the search experience, and thus still SEO."

How do I opt out of AI Overviews without leaving Google Search? Use the Search generative AI control in Search Console, available to all sites since August 31, 2026. Google says opting out stops traffic and impressions from those features and isn't used as a ranking signal for the rest of Search.

Sources

1. 01Google Search Central, "Optimizing your website for generative AI features on Google Search", published May 15, 2026, last updated July 10, 2026. https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
2. 02Google Search Central, "AI features and your website", last updated December 10, 2025. https://developers.google.com/search/docs/appearance/ai-features
3. 03Google Search Central, "Spam policies for Google web search", last updated August 28, 2026. https://developers.google.com/search/docs/essentials/spam-policies
4. 04Search Engine Land (Barry Schwartz), "Google confirms spam policies apply to AI Overviews and AI Mode", May 15, 2026. https://searchengineland.com/google-updates-search-spam-policies-to-clarify-it-applies-to-generative-ai-responses-477657
5. 05Google Crawling Infrastructure, "List of Google's common crawlers" (Google-Extended section), last updated July 14, 2026. https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
6. 06Google Search Console Help, "Search generative AI control". https://support.google.com/webmasters/answer/16908024
7. 07OpenAI, "Overview of OpenAI Crawlers". https://developers.openai.com/api/docs/bots
8. 08Cloudflare (Bryan Becker), "Have it both ways: stay discoverable in search while disallowing AI training", September 15, 2026. https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/
9. 09Search Engine Journal (Roger Montti), "Google Says LLMs.Txt Comparable To Keywords Meta Tag", April 17, 2025. https://www.searchenginejournal.com/google-says-llms-txt-comparable-to-keywords-meta-tag/544804/
10. 10Search Engine Land (Barry Schwartz), "Google doesn't want you to create bite-sized chunks of your content", January 9, 2026. https://searchengineland.com/google-doesnt-want-you-to-create-bite-sized-chunks-of-your-content-467269
11. 11Ahrefs (Louise Linehan, Xibeijia Guan), "We Tracked 1,885 Pages Adding Schema. AI Citations Barely Moved.", May 11, 2026. https://ahrefs.com/blog/schema-ai-citations/
12. 12Aggarwal et al., "GEO: Generative Engine Optimization", KDD 2024. https://arxiv.org/abs/2311.09735
13. 13Puerto et al., "C-SEO Bench: Does Conversational SEO Work?", arXiv, June 2025, revised October 20, 2025. https://arxiv.org/abs/2506.11097
14. 14Lily Ray, "Is Google Finally Cracking Down on Self-Promotional Listicles?", February 3, 2026. https://lilyraynyc.substack.com/p/is-google-finally-cracking-down-on
15. 15Ahrefs (Ryan Law), "New Study: AI Assistants Prefer to Cite 'Fresher' Content (17 Million Citations Analyzed)", July 28, 2025, updated April 27, 2026. https://ahrefs.com/blog/do-ai-assistants-prefer-to-cite-fresh-content
16. 16Bing Webmaster Blog, "Introducing AI Performance in Bing Webmaster Tools Public Preview", February 10, 2026. https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview
17. 17Vercel (Giacomo Zecchini, Alice Alexandra Moore, Malte Ubl, Ryan Siddle), "The rise of the AI crawler", December 17, 2024. https://vercel.com/blog/the-rise-of-the-ai-crawler

See what Google's AI Overview says about your brand, in the country you sell in. Nordcite's AI Visibility Report runs a query set written for your brand in the markets you choose and photographs every AI Overview: who is named, which sources are cited, dated. A person approves your queries before anything runs and reads the report before it ships. Each report is scoped and quoted for the brand it describes. See a sample report · Ask for a quote