# llms.txt Generator — Full Content > Free llms.txt generator: paste a URL, get deploy-ready llms.txt + llms-full.txt with instructions. Honest about the emerging convention. Built by AEOForged. --- # What Is llms.txt? Free Guide + Generator URL: https://llmstxtgen.com/llms-txt llms.txt is a community-proposed Markdown file placed at a site's root (/llms.txt) that gives AI models a clean, curated map of your most important content: an H1 site name, a one-line summary, and sections of links with short descriptions. A companion llms-full.txt can inline fuller content for ingestion. It was proposed by Jeremy Howard of Answer.AI in September 2024 and is documented at llmstxt.org, but it remains an unofficial convention — not a standard ratified by the W3C or IETF, and not something the major AI engines have publicly committed to consuming. Treat it as low-risk content hygiene that is cheap to publish, not as a proven lever for more citations. ## What are the key takeaways? llms.txt is a community-proposed Markdown file at /llms.txt that gives AI models a curated map of your site's most important pages. - Required format: H1 site name, optional summary, H2 sections of described links. - llms.txt complements robots.txt (access) and sitemap.xml (full URL inventory) — it is a curated highlight reel for models. - Generate a first draft with the free [llms.txt generator](/generator); review before publishing. - There is no public evidence llms.txt measurably increases AI citations as of 2026. - Fix crawl access and server-rendered HTML before investing in llms.txt. ## What is llms.txt? llms.txt is a proposed convention for a single Markdown file, served at the root of a website as /llms.txt, that points AI models and agents at the pages you most want them to read. The proposal was introduced by Jeremy Howard, co-founder of Answer.AI and fast.ai, in September 2024, and is documented at llmstxt.org. The stated motivation is that large language models work from limited context windows, so a concise, structured index of high-value pages is more useful to them than an unfiltered crawl. It is important to be precise about its status: llms.txt is an emerging community proposal. It has been adopted by a number of developer-tool and documentation sites, but it is not an official web standard and no major answer engine has publicly documented that it reads or rewards the file. That gap between adoption and proven effect is the single most important thing to understand before you invest in one. - A Markdown file at /llms.txt that maps your key content for AI models. - Proposed by Jeremy Howard (Answer.AI) in 2024; documented at llmstxt.org. - Designed around the reality that models have limited context windows. - An emerging convention, not a ratified standard or a documented ranking factor. ## What is the exact format of an llms.txt file? The llmstxt.org proposal defines a deliberately simple, Markdown-based structure so the file is readable by both humans and models. There is one required element and a small set of optional ones, which keeps the format easy to generate and to validate. At minimum the file opens with an H1 containing the site or project name. That is the only strictly required line. Below it you can add a blockquote with a short summary, free-form Markdown describing the project, and then H2 sections — typically named things like Docs, Guides, or API — each containing a Markdown list of links in the form [name](url): optional description. The companion file, llms-full.txt, follows the same idea but expands it: it inlines fuller descriptions and, for content-heavy sites, more of the actual page text, so a model can ingest substantial context in a single fetch rather than following every link. 1. Start with an H1: the site or project name (the only required line). 2. Add an optional blockquote summary: one or two sentences on what the site is. 3. Group your links under H2 sections (Docs, Guides, API, About). 4. List each link as [name](url): short description. 5. Optionally publish llms-full.txt with fuller, inlined content. ## How do you generate an llms.txt file? You can write llms.txt by hand for a small site, but generating it is faster and more consistent for anything with more than a handful of pages. The generation process is the same whether you do it manually or with a tool: enumerate your important URLs, read each page's title and description, group related pages into sections, and render the result as spec-style Markdown. Our [free generator](/generator) does this for you — you enter a URL, it discovers up to 20 of your key pages, groups them semantically into sections, and outputs both llms.txt and llms-full.txt plus deployment instructions. For larger or docs-heavy sites, many teams generate the file as part of their build so it stays in sync with the content. Whichever route you take, review the output before publishing. Generation gets you 90% of the way, but you know which pages actually matter most and which descriptions misrepresent a page — a human pass keeps the map accurate. ## How is llms.txt different from robots.txt and sitemap.xml? These three files are often confused because they all live at the site root and all relate to crawlers, but they do different jobs. robots.txt controls access — which user agents may or may not crawl which paths. sitemap.xml is an exhaustive, machine-readable inventory of URLs to help search crawlers find everything. llms.txt is neither a gate nor an exhaustive index. It is an opinionated, curated highlight reel aimed specifically at language models: a short list of the pages you most want a model to understand, with human-readable context. Where a sitemap says 'here is everything', llms.txt says 'here is what matters, and here is what each thing is'. They are complementary, not substitutes. A complete setup keeps robots.txt for access rules, sitemap.xml for full discovery, and optionally adds llms.txt as a curated map for AI consumers. None of them, on its own, makes your content the best answer — that is a content-quality problem. - robots.txt: access control — who can crawl what. - sitemap.xml: complete URL inventory for discovery. - llms.txt: a curated, described highlight reel for AI models. - They work together; llms.txt does not replace the other two. ## Does llms.txt actually improve AI visibility? There is no public, controlled evidence that publishing llms.txt measurably increases how often AI engines cite a site, and the major engines have not documented that they read it. Some have publicly downplayed dedicated AI files in favour of standard, crawlable HTML. So any claim that llms.txt 'boosts your AI rankings' is, today, unproven. What can be said honestly is that the file is low-risk and cheap. It is small, it does not interfere with normal crawling, and at worst it is ignored. For documentation sites and tools — where a clean map of canonical, versioned pages genuinely helps — it is reasonable hygiene. For a small marketing site, the upside is modest. The priority order matters: fix crawl access and rendering first (can bots fetch your pages and is the content in the served HTML?), then make each page the clearest answer to a real question, and treat llms.txt as the optional cherry on top — not the cake. When you want a deeper ranked map (up to 100 pages), install-ready AI operability files, or measured crawl and extractability checks — use [AEOForged](https://aeoforged.com). This free generator builds a ≤20-page content map; AEOForged is the next step. ## What are the key takeaways? llms.txt is a simple, useful idea with an honest caveat. It gives models a curated map of your content in a readable Markdown format, it is easy to generate and maintain, and it costs almost nothing to publish — but its effect on citations is unproven and engine support is inconsistent. - Publish llms.txt as low-risk hygiene, especially for docs and developer tools. - Keep the format spec-style: H1 name, summary, H2 sections of described links. - Use llms-full.txt to inline fuller content for one-fetch ingestion. - Do not expect a ranking or citation boost — there is no evidence of one. - Fix crawl access, rendering and content quality before worrying about this file. ## FAQ ### What is llms.txt? llms.txt is a community-proposed Markdown file at a site's root (/llms.txt) that gives AI models a curated map of your key pages: an H1 site name, a short summary, and H2 sections of links with one-line descriptions. It was proposed by Jeremy Howard in 2024 and documented at llmstxt.org. ### Is llms.txt an official standard? No. It is an emerging community proposal, not a standard ratified by the W3C or IETF, and no major AI engine has publicly documented that it reads the file. It has real adoption among developer and documentation sites, but its status is a convention, not a standard. ### Will publishing llms.txt get my site cited more by AI? There is no public, controlled evidence that it measurably increases citations, and engine support is inconsistent and undocumented. Treat it as low-risk hygiene rather than a ranking lever, and prioritize crawl access, rendering and content quality first. ### How is llms.txt different from a sitemap? A sitemap is an exhaustive list of every URL for discovery; llms.txt is a curated, described highlight reel aimed at language models. The sitemap says 'here is everything', llms.txt says 'here is what matters and what each thing is'. They are complementary. --- # llms.txt vs llms-full.txt: Which File Do You Need? URL: https://llmstxtgen.com/llms-full-txt-explained llms.txt and llms-full.txt are two files from the same proposal that serve different needs. llms.txt is a concise Markdown map — site name, summary, and sections of links with short descriptions — meant as a lightweight index. llms-full.txt follows the same structure but inlines fuller descriptions and, for content sites, much more of the actual page text, so a model can ingest substantial context in a single request instead of following every link. Publish llms.txt for almost any site; add llms-full.txt when your content is text-heavy (docs, guides, references) and you want models to absorb it without extra fetches. Neither file is a proven ranking lever — both are optional content hygiene. ## What are the key takeaways? llms.txt and llms-full.txt solve different depth problems: one is a scannable index of 10–20 links, the other inlines fuller content for one-fetch ingestion as of 2026. - llms.txt is a concise Markdown link map deployed at /llms.txt. - llms-full.txt inlines fuller descriptions and page text at /llms-full.txt. - Publish llms.txt on almost any site; add llms-full.txt when content is text-heavy (docs, guides, references). - Both files come from the 2024 community proposal at llmstxt.org — not a proven citation lever. - Generate both files free with the [llms.txt generator](/generator) from any public URL. ## What does each file contain? llms.txt is a concise index of typically 10–20 described URLs; llms-full.txt uses the same H2 sections but inlines fuller page text so a model can absorb context in one fetch, per the companion format at [llmstxt.org](https://llmstxt.org/) (September 2024). Both files share the Markdown skeleton from the [Answer.AI proposal](https://www.answer.ai/): an H1 site name, an optional blockquote summary, and H2 sections of links. llms-full.txt expands each entry with fuller descriptions and, for docs-heavy sites, substantive body text from the listed pages. - llms.txt: concise index — name, summary, sections of described links. - llms-full.txt: same structure, but with fuller descriptions and inlined page content. - Both are Markdown and both live at the site root (/llms.txt, /llms-full.txt). ## When should you publish llms-full.txt? Publish llms-full.txt when your site is text-heavy (documentation, API references, knowledge bases) and you want models to ingest up to 20 high-value pages in one fetch without following every link. For a small marketing site or portfolio, llms-full.txt adds limited value because there is not enough content to inline. Publishing only llms.txt is a reasonable, lower-maintenance choice in that case. Keep the file focused: llms-full.txt can grow large quickly, so curate genuinely high-value pages rather than dumping your entire sitemap. That keeps maintenance manageable and respects model context limits documented at [llmstxt.org](https://llmstxt.org/). ## How do you generate both files? Generation follows four steps: discover key URLs, read each page title and description, group links into logical H2 sections, and render spec-style Markdown. llms.txt outputs short pointers; llms-full.txt includes fuller text for the same URLs. Our [free generator](/generator) produces both from one public URL: it analyses up to 20 pages, groups them semantically, and returns deployment instructions. Review and trim before publishing — generation is a strong first draft, not a final answer. For a deep ranked map (up to 100 pages) continue on [AEOForged](https://aeoforged.com). Jeremy Howard of [fast.ai](https://www.fast.ai/) introduced the underlying convention in 2024. ## Which file should you publish first? Publish llms.txt first on almost every site because it is the lightweight index (typically 10–20 described URLs) defined at [llmstxt.org](https://llmstxt.org/) since September 2024. Add llms-full.txt only when your content is text-heavy enough to benefit from inlined page text — documentation sites, API references, and guide libraries are the strongest fit. Marketing sites with under 10 key pages rarely need the fuller edition. After you publish either file, the open question is whether agents can reach and extract your pages at all. [AEOForged](https://aeoforged.com) measures crawlability and extractability, and can deepen the llms.txt map beyond this free 20-page cap — the layers llms.txt alone does not replace. ## FAQ ### What is the difference between llms.txt and llms-full.txt? llms.txt is a concise Markdown index: site name, summary, and sections of links with short descriptions. llms-full.txt uses the same structure but inlines fuller descriptions and, for content sites, much more page text, so a model can ingest context in a single fetch. ### Do I need to publish both files? No. llms.txt alone is fine for most sites. Add llms-full.txt when your content is text-heavy — documentation, references, guide libraries — and you want models to absorb the substance without fetching every page individually. ### Can llms-full.txt get too big? Yes. Because it inlines page text it can grow large, so keep it focused on genuinely high-value pages rather than the whole site. That keeps it maintainable and respects the limited context a model will actually use. --- # How to Write llms.txt (Free Format Checklist) URL: https://llmstxtgen.com/how-to-write-llms-txt To write an llms.txt file, create a Markdown file with an H1 holding your site or project name (the only required element), an optional blockquote summary, and H2 sections — like Docs, Guides or API — each containing a Markdown list of links in the form [name](url): short description. Save it as llms.txt and deploy it at your site root so it resolves at /llms.txt. Keep links curated rather than exhaustive, write descriptions that say what each page actually answers, and make sure the file is valid Markdown. The most common mistakes are dumping every URL, omitting descriptions, and breaking the Markdown structure so the file can't be parsed cleanly. ## What are the key takeaways? A valid llms.txt file requires only an H1 site name; the [llmstxt.org spec](https://llmstxt.org/) recommends 3–5 H2 sections with 10–20 curated links for most sites in 2026. - Required: one H1 with your site or project name. - Recommended: blockquote summary, H2 sections, and - [name](url): description link lists. - Deploy at your site root so the file resolves at /llms.txt. - Curate 10–20 high-value pages — not your full sitemap. - Use the free [generator](/generator) for a spec-style first draft, then review before publishing. ## What are the steps to write a valid llms.txt? Writing a valid llms.txt takes six steps: add an H1 site name (the only required line per [llmstxt.org](https://llmstxt.org/)), write a blockquote summary, group links under H2 sections, describe each link, validate Markdown syntax, and deploy at /llms.txt. After the H1, add a one- or two-sentence blockquote summarizing what the site is and who it serves. Create H2 sections (Docs, Guides, API) and list links as `- [name](url): short description`. Save as plain Markdown named llms.txt at your site root. 1. Create a Markdown file named llms.txt. 2. Add an H1 with the site or project name (required). 3. Add a one- or two-sentence blockquote summary (optional but recommended). 4. Create H2 sections that group links by purpose (Docs, Guides, API). 5. Under each section, list links as - [name](url): short description. 6. Validate that it's clean Markdown, then deploy at the site root. ## What does a good llms.txt look like? A good llms.txt is a curated table of contents: site name and summary at the top, 3–5 logical H2 sections, and descriptions that state what question each page answers — typically 10–20 links total, not your full sitemap. Curation is the differentiator introduced in the [2024 Answer.AI proposal](https://www.answer.ai/). Ten well-described, high-value links beat 100 bare URLs for both model context limits and long-term maintainability. - Site name and a clear one-line summary at the top. - Logical H2 sections, not one giant flat list. - Descriptions that state what each page answers. - Curated, high-value links rather than your entire sitemap. ## What are the most common llms.txt mistakes? The three most common llms.txt mistakes are sitemap dumps (every URL, no descriptions), invalid Markdown (broken link syntax or inconsistent heading levels), and stale links after pages move or redirect. Other frequent issues in 2026: linking to pages crawlers cannot reach, overstating citation impact to stakeholders, and skipping human review after automated generation. Keep the file accurate, valid, and honest about what the [llmstxt.org spec](https://llmstxt.org/) actually promises. A clean file still does not prove bots can fetch your HTML. After you fix the map, check crawl access and extractability on [AEOForged](https://aeoforged.com) — and run a deep ranked scan when 20 pages is not enough. That is the measurement layer behind this free generator. - Dumping every URL with no descriptions (it's a map, not a sitemap). - Invalid or inconsistent Markdown that won't parse cleanly. - Letting links rot as pages move or get renamed. - Linking to pages crawlers can't actually reach. - Believing — or claiming — it guarantees more AI citations (0% controlled evidence as of 2026). ## How do you validate llms.txt before deploy? Validate llms.txt by checking three things before deploy: the file resolves at /llms.txt, every link returns HTTP 200, and the Markdown parses cleanly with one H1 and valid `- [name](url): description` lists. Run a quick link check after each edit — broken links undermine the map faster than stale descriptions. The [llmstxt.org format](https://llmstxt.org/) requires only the H1; everything else is optional but recommended for model usability. ## FAQ ### What is the only required part of an llms.txt file? An H1 containing the site or project name. Everything else — the summary blockquote, the H2 sections and the link lists — is optional in the spec, though sections and descriptions are what make the file actually useful to a model. ### Where do I deploy the llms.txt file? At your site root, so it resolves at https://yourdomain.com/llms.txt. How you do that depends on your stack — a static file in your public directory, or a route that returns the Markdown with a text/plain content type. ### Should llms.txt list every page on my site? No. It's a curated highlight reel, not a sitemap. List the high-value pages you most want a model to understand, with a clear description each. A short, well-described file beats an exhaustive, bare one. --- # llms.txt for Docs Sites: Keep the Map Fresh URL: https://llmstxtgen.com/llms-txt-for-documentation-sites Documentation sites benefit most from llms.txt because their value is structured, text-heavy and frequently referenced by AI coding assistants — exactly the content a curated map and an inlined llms-full.txt help a model use. To do it well, organize the file around how developers navigate docs (getting started, guides, API reference, changelog), describe each link by the task it solves, and generate the file as part of your build so it stays in sync as docs change. It remains an emerging convention with no proven ranking effect, so treat it as useful developer-facing hygiene rather than a guaranteed visibility win. ## What are the key takeaways? Documentation sites are the strongest fit for llms.txt because docs are structured, text-heavy, and frequently queried through AI coding assistants in 2026. - Structure docs maps as Getting Started, Guides, API Reference, and Changelog — task-oriented H2 sections. - Describe each link by the task it solves, not just the page title. - Generate llms.txt from your docs build/CI so it stays in sync — hand-maintained files rot quickly. - llms-full.txt is especially valuable for API reference content that models need in one fetch. - There is no published proof llms.txt increases AI citations — treat it as low-cost developer hygiene. ## Why do documentation sites benefit most from llms.txt? Documentation sites are the highest-value llms.txt use case because docs are dense, versioned, and often the exact content developers ask AI coding assistants about in 2026. Developer-tool and documentation sites were among the first to publish llms.txt after the [September 2024 proposal](https://llmstxt.org/) from Jeremy Howard of [Answer.AI](https://www.answer.ai/). Their content maps cleanly onto Getting Started, Guides, API, and Changelog sections. There is no published controlled proof that llms.txt increases AI citations. For docs the cost is low ($0 to publish) and the structure is a natural fit, so shipping a file is reasonable hygiene. ## How should you structure llms.txt for docs? Structure docs llms.txt with four task-oriented H2 sections — Getting Started, Guides, API Reference, Changelog — and describe each link by the developer task it solves, not just the page title. Link canonical, current pages only. For versioned docs, point at the authoritative version and omit deprecated URLs. An llms-full.txt that inlines core API reference content is especially valuable because it spares models from fetching every endpoint page individually. - Use task-oriented sections: Getting Started, Guides, API, Changelog. - Describe links by the task they solve, not just the page title. - Link canonical, current (and correctly versioned) pages only. - Use llms-full.txt to inline core reference content for one-fetch context. ## How do you keep llms.txt in sync as docs change? Keep docs llms.txt in sync by generating it from your build or CI pipeline so new pages appear and removed pages disappear automatically — hand-maintained files drift within weeks on active doc sites. If you regenerate on demand instead, schedule a quarterly review at minimum: validate Markdown, check links, and trim stale descriptions. The goal is that the file reflects live docs, not a snapshot from launch day. See [fast.ai](https://www.fast.ai/) tooling sites for examples of docs-first adoption patterns. 1. Generate llms.txt/llms-full.txt from your docs source, not by hand. 2. Wire generation into your build or CI so it updates with the docs. 3. Validate Markdown and check links on each regeneration. 4. Do a periodic human review to keep curation and descriptions accurate. ## What tools generate llms.txt for documentation sites? Documentation teams generate llms.txt with three approaches: CI/build plugins that emit the file from docs source, on-demand generators that analyse up to 20 live URLs, or hand-authored Markdown for small sites. Our [free generator](/generator) discovers pages from a public docs URL and outputs both llms.txt and llms-full.txt — useful for a first draft before you wire generation into your docs pipeline. Jeremy Howard of [Answer.AI](https://www.answer.ai/) introduced the underlying format in September 2024. Pair the map with measured crawlability: docs that hide behind client-only rendering still fail agents. [AEOForged](https://aeoforged.com) is built for those checks and for deeper ranked maps — this site stays focused on the free ≤20-page draft. ## FAQ ### Why is llms.txt a good fit for documentation sites? Docs are structured, text-heavy and frequently queried through AI coding assistants, so a curated map plus an inlined llms-full.txt gives models compact, on-topic context. The content also maps cleanly onto sections like getting started, guides and API reference. ### How do I keep llms.txt up to date for changing docs? Generate it from your docs source as part of your build or CI pipeline so it updates automatically, validate the Markdown and links on each run, and do a periodic human review. Hand-maintained files drift out of date quickly. ### Does llms.txt help my docs rank in AI answers? There's no published evidence that it increases citations, and engine support is undocumented. For docs the cost is low and the structure is a natural fit, so it's reasonable hygiene — but treat it as developer-facing housekeeping, not a guaranteed ranking win. --- # How Often to Update llms.txt (Without Busywork) URL: https://llmstxtgen.com/llms-txt-maintenance-schedule Regenerate llms.txt when you ship major new sections (products, docs, pricing) or remove flagship URLs that still appear in the file; otherwise review the map at least quarterly on active sites. Small typo fixes rarely require a full rebuild — patch the relevant H2 block in llms-full.txt and bump dateModified metadata on pages that reference the file. There is no public controlled evidence that updating llms.txt alone increases AI citations as of 2026; treat maintenance as inexpensive hygiene aligned with the [llmstxt.org](https://llmstxt.org/) proposal from Jeremy Howard of [Answer.AI](https://www.answer.ai/), not a guaranteed visibility lever. ## What are the key takeaways? llms.txt maintenance is about accuracy, not frequency: regenerate after major IA changes, patch after minor edits, and review quarterly (every 90 days) on active sites. - Full regeneration: new product/docs areas, pricing tiers, or removed flagship URLs. - Partial update: typo fixes, single new guides — edit llms-full.txt in place. - Review cadence: quarterly minimum for sites that ship weekly; align slower sites with major releases. - No proven citation boost — maintenance is $0-risk housekeeping per the 2024 community proposal. - Use the [free generator](/generator) after big releases, then human-review before deploy. ## When should you regenerate the whole llms.txt file? Regenerate the entire llms.txt when your information architecture shifts enough that the curated list no longer matches what you would hand a human editor — for example a new docs section, a pricing tier, or redirects that leave dead URLs in the old file. Teams that generate llms.txt in CI should wire regeneration into the same pipeline so the file tracks every deploy automatically. Manual teams should run the [generator](/generator) after major releases and diff against the live /llms.txt before publishing. - New product, docs, or pricing areas that AI assistants should understand. - Removed or redirected URLs still listed in the old map. - Sitemap changes that alter which pages are canonical. ## When is a partial llms.txt update enough? A partial llms.txt update is enough when fewer than 20% of your curated links change — for example one new guide, a headline fix, or a single renamed URL in a 15-link file. Edit the relevant H2 block in llms-full.txt and bump dateModified in page JSON-LD. Avoid regenerating on every commit if your 10–20 link map still represents the site accurately. ## What maintenance cadence works for most teams? A quarterly review cycle means four maintenance passes per year for sites that ship weekly; slower teams can align reviews with major releases instead. Each review validates Markdown syntax, checks links return HTTP 200, and confirms descriptions match page intent. Keep robots.txt and sitemap.xml healthy first — the curated map sits on top of crawlable HTML. 1. After each major release, run the generator and diff against the live file. 2. Quarterly: scan for dead links and stale descriptions. 3. When only one section changed, patch llms-full.txt and bump page dates. 4. Validate robots.txt access before investing in llms.txt maintenance. ## Does updating llms.txt improve AI citations? There is no public controlled evidence that updating llms.txt measurably increases AI citations as of 2026 — 0 published studies confirm a citation lift from maintenance alone. Maintenance still matters for accuracy: a stale map misleads any agent that does consume it. Treat updates as low-cost developer hygiene from the [Answer.AI proposal](https://www.answer.ai/) — not a proven ranking lever. If you want a measured read on whether AI can use the live site after a refresh — or a deep ranked remap beyond 20 pages — continue on [AEOForged](https://aeoforged.com). Crawlability and extractability sit upstream of any content map. ## What is the minimum review checklist? The minimum llms.txt review checklist has four items: confirm /llms.txt resolves, verify all links return 200, validate Markdown syntax, and confirm descriptions match current page intent. Run this checklist quarterly on active sites or after every major release. The entire review takes under 30 minutes for a typical 15-link file — far less than maintaining a full sitemap narrative. ## FAQ ### How often should I update llms.txt? Regenerate when you add major new sections or remove flagship URLs; otherwise review at least quarterly. Small edits can be patched in llms-full.txt without a full rebuild. ### Does updating llms.txt improve AI citations? There is no public controlled evidence that it does as of 2026. Treat updates as inexpensive hygiene, not a proven ranking or citation lever. ### Should I regenerate llms.txt on every deploy? No. Regenerate when information architecture changes materially. Routine typo fixes and single new pages can be patched in llms-full.txt without rebuilding the entire map. --- # llms.txt vs robots.txt: What Each File Does URL: https://llmstxtgen.com/llms-txt-vs-robots-txt robots.txt tells crawlers which paths they may fetch; llms.txt is an optional Markdown map of the pages you most want language models to understand. They live at the same site root but solve different problems — access control versus a curated highlight reel. Keep robots.txt healthy first so agents can reach your HTML; add llms.txt only as optional hygiene. There is no public evidence that llms.txt replaces robots.txt or increases AI citations on its own. ## What are the key takeaways? robots.txt and llms.txt both sit at the site root, but one gates crawl access and the other offers a curated map for models — they are complementary, not substitutes. - robots.txt: allow/disallow rules for user agents (including AI bots). - llms.txt: Markdown H1 + sections of described links for AI consumers. - sitemap.xml is a third file — a full URL inventory for discovery. - Fix robots and crawlable HTML before investing in llms.txt. - Generate a draft map with the [free generator](/generator); review before deploy. ## What does robots.txt control? robots.txt is a long-standing access-control file at /robots.txt. It lists which user agents may crawl which path prefixes. Search crawlers and many AI bots respect it as a polite convention — though enforcement varies by vendor. If a bot is disallowed from your docs or product pages, no content map at /llms.txt can fix that. Access comes first; curation is optional. - Controls who may fetch which paths. - Does not describe what each page is about. - Does not guarantee indexing or citations. ## What does llms.txt add that robots.txt does not? llms.txt, proposed at [llmstxt.org](https://llmstxt.org/) by Jeremy Howard of [Answer.AI](https://www.answer.ai/) in September 2024, is a curated Markdown index: site name, short summary, and H2 sections of links with one-line descriptions. Where robots.txt says “you may crawl /docs”, llms.txt says “start with Getting Started, then the API overview, then pricing — here is what each answers.” It is neither a gate nor a full sitemap. ## Do you need both files? Yes for a complete setup: keep robots.txt accurate for every agent you care about, keep sitemap.xml for discovery, and optionally add llms.txt as a highlight reel for models. Skipping robots.txt while publishing llms.txt is backwards. Skipping llms.txt while robots and HTML are healthy is fine — the map is optional hygiene, not a ranking requirement. After access works, measure whether agents can extract answers from live pages on [AEOForged](https://aeoforged.com). This site only generates the content map. ## How do you generate llms.txt after robots is healthy? Confirm /robots.txt allows the pages you want mapped, then run the [free llms.txt generator](/generator) on your public URL. Review the curated list, trim noise, and deploy both /llms.txt and /llms-full.txt at the site root. See also the pillar guide on [what llms.txt is](/llms-txt) and [how to write one by hand](/how-to-write-llms-txt). 1. Validate robots.txt allows key paths for AI bots you care about. 2. Generate a draft with the free generator (or write Markdown by hand). 3. Human-review links and descriptions before publishing. 4. Deploy at https://yourdomain.com/llms.txt (and optionally llms-full.txt). ## FAQ ### Is llms.txt a replacement for robots.txt? No. robots.txt controls crawl access; llms.txt is an optional curated content map. You need accurate robots rules before a content map can help any agent that respects them. ### Does llms.txt tell AI bots they can crawl my site? No. Allow/disallow belongs in robots.txt (and related headers/meta). llms.txt only points at pages you want models to understand if they can already fetch them. ### Should I list the same URLs in robots.txt and llms.txt? Not the same job. robots.txt uses path rules; llms.txt lists a short curated set of important pages with descriptions. Keep them consistent — do not map pages your robots file forbids. --- # llms.txt for Next.js (App Router Deploy) URL: https://llmstxtgen.com/llms-txt-for-nextjs On Next.js App Router, the simplest way to publish llms.txt is to place llms.txt and llms-full.txt in the public/ folder so they resolve at /llms.txt and /llms-full.txt on every deploy. Alternatively, add app/llms.txt/route.ts (and llms-full.txt) that returns text/plain Markdown. Generate a first draft with a free generator, review it, commit the files, and redeploy. llms.txt remains an emerging convention — not a proven citation lever. ## What are the key takeaways? Next.js App Router sites can ship llms.txt as static files in public/ or as Route Handlers that return text/plain — both resolve at the site root after deploy. - Easiest path: public/llms.txt + public/llms-full.txt. - Dynamic path: app/llms.txt/route.ts returning Markdown with Content-Type: text/plain. - Generate a draft with the [free generator](/generator), then commit reviewed files. - Keep robots.txt and crawlable HTML healthy first. - No published proof that llms.txt alone increases AI citations. ## How do you deploy llms.txt from the public folder? Create public/llms.txt and public/llms-full.txt in your Next.js project. Anything in public/ is served from the site root, so production will expose https://yourdomain.com/llms.txt after the next deploy. This matches how many docs and marketing sites publish the [llmstxt.org](https://llmstxt.org/) proposal: static Markdown, no runtime work. 1. Generate or write valid Markdown for both files. 2. Save them as public/llms.txt and public/llms-full.txt. 3. Deploy (Vercel, Node host, or your usual pipeline). 4. Confirm the live URLs return 200 and text/plain or text/markdown. ## When should you use a Route Handler instead? Use a Route Handler when the map should be generated at build or request time from a content source — for example a CMS or MDX inventory — so you do not hand-edit Markdown on every content change. A minimal handler returns new Response(markdown, { headers: { 'Content-Type': 'text/plain; charset=utf-8' } }) from app/llms.txt/route.ts. Prefer caching or build-time generation over regenerating on every cold request for large sites. ## How do you generate the Markdown for a Next.js site? Paste your production URL into the [free llms.txt generator](/generator). It discovers up to 20 pages, groups them into sections, and returns both files plus deployment notes. Review titles and descriptions before you commit — generation is a draft. Larger App Router sites can continue on [AEOForged](https://aeoforged.com) for a deep ranked scan. For docs-heavy App Router sites, also read [llms.txt for documentation sites](/llms-txt-for-documentation-sites) and wire regeneration into CI when the docs source of truth changes. ## What should you measure after deploy? Confirm /llms.txt and /robots.txt resolve, then check whether AI bots can fetch and extract your important pages. Publishing a map does not fix client-only shells or blocked paths. [AEOForged](https://aeoforged.com) measures crawlability and extractability and can produce install-ready AI operability files after a deeper map. This free site stays focused on the ≤20-page draft. ## FAQ ### Where do I put llms.txt in a Next.js App Router project? The simplest option is public/llms.txt (and public/llms-full.txt) so Next serves them at the site root. You can also add Route Handlers under app/llms.txt/route.ts that return text/plain Markdown. ### Does Vercel need special config for llms.txt? No special config for static files in public/. After deploy, open https://yourdomain.com/llms.txt and confirm a 200 response. Route Handlers work like any other App Router route. ### Will llms.txt improve my Next.js site's AI rankings? There is no public controlled evidence that it does. Treat it as optional hygiene after robots.txt and crawlable HTML are correct — not a guaranteed ranking or citation lever. --- # llms.txt for WordPress: Root File Options URL: https://llmstxtgen.com/llms-txt-wordpress WordPress serves the site root from the web host’s document root (often public_html), not from the WordPress admin media library. To publish llms.txt, upload llms.txt and optionally llms-full.txt next to wp-config.php / index.php so they resolve at /llms.txt, or use a plugin/redirect that serves Markdown at those paths. Generate a draft with a free generator, review it, then upload. llms.txt is an emerging convention — not a proven citation boost. ## What are the key takeaways? On WordPress, llms.txt must resolve at the domain root — usually by uploading files into the host document root alongside WordPress core files, not as a Media Library attachment. - Upload llms.txt (and llms-full.txt) via FTP, SFTP, or hosting file manager. - Confirm https://yourdomain.com/llms.txt returns the Markdown body. - Do not bury the file under /wp-content/uploads/ if you want the conventional root URL. - Generate a draft with the [free generator](/generator); human-review before upload. - No published proof that WordPress + llms.txt alone increases AI citations. ## Where does the file go on a typical WordPress host? Place llms.txt in the same directory as WordPress’s index.php — commonly public_html, www, or htdocs depending on the host. That is the document root browsers hit for https://yourdomain.com/. The Media Library stores uploads under /wp-content/uploads/, which is the wrong location for the [llmstxt.org](https://llmstxt.org/) convention that expects /llms.txt at the root. 1. Generate and review Markdown for llms.txt (and optionally llms-full.txt). 2. Connect via FTP/SFTP or open the host file manager. 3. Upload both files into the WordPress document root. 4. Load /llms.txt in a browser and verify the content. ## What if the host or security plugin blocks root uploads? Some managed WordPress hosts restrict root file writes. Options then include: a small custom plugin that serves Markdown at /llms.txt, a rewrite rule that maps /llms.txt to a static file elsewhere, or asking host support to place the file for you. Whatever path you choose, the public URL should still be /llms.txt with a text body — not an HTML theme template wrapped around the Markdown. ## How do you keep the WordPress map up to date? Regenerate after major site structure changes — new product or docs sections, removed flagship pages — and review at least quarterly on active sites. See [how often to update llms.txt](/llms-txt-maintenance-schedule). If you publish many posts weekly, curate only the evergreen money and docs pages in the map; do not dump every blog URL. ## What should you check besides the file? Confirm robots.txt does not block AI bots you care about, and that important pages return crawlable HTML (not empty client shells). A root Markdown map cannot fix blocked or empty pages. For measured crawlability and extractability after you upload, use [AEOForged](https://aeoforged.com). This site generates the content map only. ## FAQ ### Can I upload llms.txt through the WordPress Media Library? You can store a file there, but the conventional URL is /llms.txt at the site root. Media Library URLs live under /wp-content/uploads/, so prefer FTP/file manager upload to the document root (or a rewrite that exposes /llms.txt). ### Does a WordPress SEO plugin create llms.txt automatically? Plugin support varies and changes over time. Check your plugin’s current docs. If it does not emit root /llms.txt, upload static files or a small serving plugin yourself. ### Will llms.txt help my WordPress posts rank in ChatGPT? There is no public controlled evidence that publishing llms.txt increases AI citations. Treat it as optional hygiene after access and content quality — not a guaranteed ranking lever.