robots.txt tells crawlers which paths they may fetch; llms.txt is an optional Markdown map of the pages you most want language models to understand. They live at the same site root but solve different problems — access control versus a curated highlight reel. Keep robots.txt healthy first so agents can reach your HTML; add llms.txt only as optional hygiene. There is no public evidence that llms.txt replaces robots.txt or increases AI citations on its own.
What are the key takeaways?
robots.txt and llms.txt both sit at the site root, but one gates crawl access and the other offers a curated map for models — they are complementary, not substitutes.
- robots.txt: allow/disallow rules for user agents (including AI bots).
- llms.txt: Markdown H1 + sections of described links for AI consumers.
- sitemap.xml is a third file — a full URL inventory for discovery.
- Fix robots and crawlable HTML before investing in llms.txt.
- Generate a draft map with the free generator; review before deploy.
What does robots.txt control?
robots.txt is a long-standing access-control file at /robots.txt. It lists which user agents may crawl which path prefixes. Search crawlers and many AI bots respect it as a polite convention — though enforcement varies by vendor.
If a bot is disallowed from your docs or product pages, no content map at /llms.txt can fix that. Access comes first; curation is optional.
- Controls who may fetch which paths.
- Does not describe what each page is about.
- Does not guarantee indexing or citations.
What does llms.txt add that robots.txt does not?
llms.txt, proposed at llmstxt.org by Jeremy Howard of Answer.AI in September 2024, is a curated Markdown index: site name, short summary, and H2 sections of links with one-line descriptions.
Where robots.txt says “you may crawl /docs”, llms.txt says “start with Getting Started, then the API overview, then pricing — here is what each answers.” It is neither a gate nor a full sitemap.
Do you need both files?
Yes for a complete setup: keep robots.txt accurate for every agent you care about, keep sitemap.xml for discovery, and optionally add llms.txt as a highlight reel for models.
Skipping robots.txt while publishing llms.txt is backwards. Skipping llms.txt while robots and HTML are healthy is fine — the map is optional hygiene, not a ranking requirement.
After access works, measure whether agents can extract answers from live pages on AEOForged. This site only generates the content map.
How do you generate llms.txt after robots is healthy?
Confirm /robots.txt allows the pages you want mapped, then run the free llms.txt generator on your public URL. Review the curated list, trim noise, and deploy both /llms.txt and /llms-full.txt at the site root.
See also the pillar guide on what llms.txt is and how to write one by hand.
- 1Validate robots.txt allows key paths for AI bots you care about.
- 2Generate a draft with the free generator (or write Markdown by hand).
- 3Human-review links and descriptions before publishing.
- 4Deploy at https://yourdomain.com/llms.txt (and optionally llms-full.txt).
Frequently asked questions
Is llms.txt a replacement for robots.txt?+
No. robots.txt controls crawl access; llms.txt is an optional curated content map. You need accurate robots rules before a content map can help any agent that respects them.
Does llms.txt tell AI bots they can crawl my site?+
No. Allow/disallow belongs in robots.txt (and related headers/meta). llms.txt only points at pages you want models to understand if they can already fetch them.
Should I list the same URLs in robots.txt and llms.txt?+
Not the same job. robots.txt uses path rules; llms.txt lists a short curated set of important pages with descriptions. Keep them consistent — do not map pages your robots file forbids.