How to Add llms.txt to Your Website (2026 Step-by-Step Guide)
llms.txt is a plain-text file you serve at /llms.txt that gives large language models (ChatGPT, Perplexity, Claude, Gemini) a curated map of the most important content on your site. Think of it as robots.txt for AI assistants, except instead of telling them what NOT to read, you're telling them what's most worth reading. The format was proposed by Jeremy Howard in 2024 and is increasingly recognized by AI crawlers.
What you'll need before you start
- •A live website you can serve static files from
- •Ability to upload a text file at the root of your domain (or a CMS that lets you serve custom content at a custom URL)
- •Roughly 30 minutes for a thoughtful first version
How to add llms.txt on Any Website: Step-by-step
- 1
Decide what content to highlight
Start by listing the 10-30 most important pages on your site - the ones you'd actually want an AI assistant to cite when answering a question about your category. Common starters: your homepage, a 1-2 sentence "what we do" summary, your most-trafficked feature pages, your top 5-10 blog posts, your pricing page, your API docs (if you have them), your FAQ. Skip pages that are duplicative, transactional (cart, checkout), or low-quality.
- 2
Write the llms.txt file in the proposed format
The format is simple Markdown, plus a couple of conventions. Start with an H1 of your site name. Optionally a blockquote with a 1-2 sentence summary. Then sections (H2) grouping related content, with bulleted links. Example:
# MeasureBoard > AI-native website analytics platform with a dedicated AI analyst. > Connects to Google Analytics 4, generates AI-powered reports, and > tracks how often AI search engines cite your domain. ## Product - [Features](https://www.measureboard.com/#features): Overview of every feature - [Pricing](https://www.measureboard.com/#pricing): Plans starting at free ## Guides - [What is GEO](https://www.measureboard.com/blog/what-is-geo): Generative Engine Optimization explained - [llms.txt Guide](https://www.measureboard.com/blog/llms-txt-guide): How to write an effective llms.txt ## Optional - [Compare](https://www.measureboard.com/compare): How we differ from competitors - 3
Use sections meaningfully
Standard section names that AI crawlers recognize: Product, Documentation, Guides, API, Examples, Optional. The Optional section is special - content listed there is treated as supplementary rather than essential. Use it for "nice to have" links so AI crawlers don't bloat their context window.
- 4
Add 1-line descriptions to every link
Each bulleted link should follow the format
- [Page title](URL): One-sentence description. The description matters: AI crawlers use it to decide whether to actually fetch the page or just remember the summary. Keep descriptions factual and specific - say what the page contains, not how great it is. - 5
Save as plain text, named exactly llms.txt
The file must be named
llms.txt(lowercase, no extension trick) and must be served at the root of your domain (https://yourdomain.com/llms.txt). Content type should betext/plain; charset=utf-8. Any HTTP server can serve a plain text file at a fixed path - you don't need any special tooling. - 6
Upload to your domain root
The exact path depends on your hosting. Static hosting (Netlify, Vercel, Cloudflare Pages, GitHub Pages): place
llms.txtin yourpublic/orstatic/directory and deploy. WordPress: SFTP into your site and upload to the public root, OR use a plugin that lets you serve custom files at custom paths. Shopify, Squarespace, Wix: hosted platforms generally don't allow root-level static files. Use a redirect or a sub-path workaround (covered in the platform-specific guides). - 7
Verify the file is being served correctly
Open
https://yourdomain.com/llms.txtin your browser. You should see the plain text content. Check the response headers in your browser's DevTools Network tab - the Content-Type should betext/plain(nottext/html). MeasureBoard's free Agent Readiness checker tests llms.txt presence and format alongside other AI-citability signals.
How to verify your setup is working
The simplest check: visit https://yourdomain.com/llms.txt in your browser. The content should be plain text matching what you wrote. Use DevTools (Network tab) to confirm Content-Type: text/plain. There's no Search Console equivalent for llms.txt - because it's an AI-crawler signal, not a Google one. To validate format, paste your file into MeasureBoard's Agent Readiness tool which checks structure plus other AI-readiness signals.
Common issues and fixes
Visiting /llms.txt redirects to my homepage or returns a 404
Most often this is because your CMS / framework doesn't have a route handler for arbitrary root files. Next.js: place llms.txt in the public/ directory. Hugo / Jekyll / Astro: in static/. WordPress: upload via SFTP to the public root. For server-side platforms (Express, Rails, Django), add a static-file route mapping /llms.txt to your file. Hosted platforms (Shopify, Squarespace, Wix) generally don't allow root-level files - use a workaround.
llms.txt downloads as a file instead of displaying in the browser
Your server is serving the file with a generic Content-Type: application/octet-stream instead of text/plain. Add a server config rule: in nginx location = /llms.txt { default_type text/plain; }; in Apache AddType text/plain .txt; in Cloudflare Pages, the _headers file with /llms.txt\n Content-Type: text/plain; charset=utf-8.
AI assistants still don't seem to cite my site after adding llms.txt
llms.txt is a strong signal but not a guarantee. AI assistants prioritize sites with high authority and structured data. Pair llms.txt with: comprehensive schema markup (Organization, Article, Product), authoritative content on the topic, and external citations from third-party sources. Citation in AI answers also takes time to build - track it monthly with MeasureBoard's AI Rank Tracker rather than checking daily.
Frequently asked questions
Is llms.txt required by Google or AI crawlers?
No - llms.txt is a community-proposed format (introduced by Jeremy Howard in 2024) and not yet a formal standard. AI crawlers from Anthropic, Perplexity, and OpenAI are increasingly recognizing it. It's not required, but adding one is one of the highest-leverage low-effort moves you can make for AI search visibility.
What's the difference between llms.txt and robots.txt?
robots.txt tells crawlers what NOT to access (deny lists, crawl-delay rules). llms.txt tells AI crawlers what IS most important to read (allow-lists with descriptions and structure). They serve different purposes; you should have both.
Should I add every page on my site to llms.txt?
No - the value comes from curation. List the 10-30 pages an AI assistant should actually fetch and cite. Including every URL makes the file noisy and dilutes the signal. Use the Optional section for nice-to-have links you don't want to clutter the main sections.
How do I add llms.txt to a hosted platform like Shopify or Squarespace?
Hosted platforms typically don't allow root-level static files. Workarounds: (1) Use a Cloudflare Worker / Page Rule that intercepts /llms.txt and serves your custom content. (2) Use a sub-path equivalent - e.g., publish a markdown page at /about/llms and submit that URL to AI crawler discovery channels. (3) For Shopify, you can use the shop.json + custom routes via a custom theme - but this is more involved than most stores want. The Cloudflare path is the cleanest cross-platform solution.
How long is a typical llms.txt file?
The proposed spec recommends keeping it concise enough to fit in an LLM's context window without overwhelming it - typically under 200 lines. For most sites, 30-80 lines is plenty. Sites with extensive documentation may go longer, but the signal-to-noise ratio matters more than length.
Where can I see examples of well-written llms.txt files?
Anthropic's docs site has one at docs.anthropic.com/llms.txt. The original proposal at llmstxt.org includes the format spec plus a directory of sites that have implemented it. MeasureBoard's own llms.txt at measureboard.com/llms.txt is also a working example.
Related Guides
Once you're set up, MeasureBoard turns the data into action
MeasureBoard connects to your analytics, search, and SEO tooling and generates AI-powered weekly reports, GEO readiness scores, and a dedicated AI analyst that answers questions in plain English. Free to start, connects in 30 seconds.
Get Started Free