What an llms.txt file does and why it matters

An llms.txt file is a plain text document you place in your website's root directory — the same level as your homepage — that tells AI language models what your site contains and how they should use it. When an AI system crawls the web looking for content to learn from, it checks for llms.txt the same way it checks for robots.txt, which controls what search engines can access.

The file itself is straightforward: it contains instructions, links to your content, and rules about how AI models can use what they find. Without it, large language models may still discover your site through general web crawling, but they have no clear signal about your preferences. With it, you're explicitly saying "here's what I want indexed, here's how to use it, and here's what's off-limits."

This matters because AI models train on internet text, and the way they discover and use your content affects how your writing appears in their outputs. An llms.txt file gives you a voice in that process instead of leaving it entirely to chance.

Key Takeaways

  • An llms.txt file is a text document placed in your website's root directory that tells AI models what content they can index and how to use it.
  • The file uses a straightforward format: you list allowed paths, disallowed paths, and optional metadata about your site's purpose and licensing.
  • Without an llms.txt file, AI models may still crawl your site, but you have no control over what they take or how they use it.
  • Creating one takes less than an hour and requires only a text editor and basic knowledge of your site's structure.

How llms.txt files work technically

The llms.txt standard is still emerging, but the basic structure follows a pattern similar to robots.txt. You create a plain text file with no special formatting, name it llms.txt, and upload it to the root of your domain — so it lives at yoursite.com/llms.txt, not in a subfolder.

Inside the file, you write directives that tell AI crawlers which parts of your site they can index. A typical file might start with metadata about your site: its name, description, and your contact information. Then it lists allowed paths (the pages and content you want indexed) and disallowed paths (the pages you don't). You can also specify licensing terms — whether your content can be used commercially, whether attribution is required, and so on.

When a responsible AI model crawls your site, it looks for llms.txt before indexing. If it finds one, it follows the rules inside. If it doesn't find one, the model's behavior depends on its own design — some will crawl everything, some will be cautious, and some will skip your site entirely.

What to include in your llms.txt file

Start with a header section that identifies your site. Include your site name, a brief description of what you publish, and an email address where AI developers can contact you with questions. This section doesn't control indexing — it's informational.

Next, add your allow and disallow rules. If you want all content indexed, you might write "Allow: /" to permit everything. If you want to exclude certain sections — like your admin panel, user accounts, or draft posts — you list those: "Disallow: /admin/" or "Disallow: /user-accounts/". You can be as specific as needed: "Disallow: /blog/draft-posts/" blocks only that folder, while "Disallow: /*.pdf" blocks all PDF files across your entire site.

Then specify your licensing terms. Common options include whether your content can be used for commercial purposes, whether the AI model must attribute you when it uses your text, and whether derivatives (modified versions of your work) are allowed. You might write "Commercial: false" if you want to restrict commercial use, or "Attribution: true" if you require credit.

Finally, you can add optional fields like your preferred citation format, links to your privacy policy, or your content update frequency. Keep the file straightforward — every line should serve a real purpose.

Step-by-step: creating and uploading your file

Open a plain text editor — Notepad on Windows, TextEdit on Mac, or any code editor works. Do not use Word or Google Docs, because they add invisible formatting that breaks the file.

Write your header first. It might look like this:

Name: Housing Basics Description: Free guides about housing, rental information, and tenant rights Contact: hello@housingbasics.org

Then add your allow and disallow rules. For a typical website, you might allow everything and disallow only sensitive areas:

Allow: / Disallow: /admin/ Disallow: /user-accounts/ Disallow: /private/

Add your licensing terms next:

Commercial: false Attribution: true Derivatives: false

Save the file as "llms.txt" — exactly that name, no other extension. Then upload it to your web server's root directory using FTP, your hosting control panel, or whatever method you normally use to upload files. Once it's live, visit yoursite.com/llms.txt in your browser to confirm it's there and readable.

The difference between llms.txt and robots.txt

Both files sit in your root directory and control what automated systems can access, but they serve different purposes. robots.txt controls search engine crawlers like Google and Bing — it tells them which pages to index for search results. llms.txt controls AI language models — it tells them which content they can use for training.

A search engine crawler respects robots.txt because it wants to follow your wishes and because search engines have built indexing into their business model. An AI model respects llms.txt because its developers chose to build that respect in — but not all AI models check for it yet. Some large models do; others ignore it entirely. Over time, as the standard becomes more common, more models will check for it.

You can have different rules in each file. You might allow search engines to index everything (robots.txt: Allow: /) but restrict AI models to only your published articles (llms.txt: Allow: /articles/, Disallow: /). Or you might block search engines from indexing your blog drafts but allow AI models to see them. The two files are independent.

What happens after you create your llms.txt file

Once your file is live, responsible AI models will begin checking it when they crawl your site. This doesn't happen when ready — it depends on how often each model's crawler visits your domain. Some crawl weekly, some monthly, and some only when you notify them.

You can speed up the process by telling AI developers about your file. Some models publish contact information or submission forms where you can notify them that your llms.txt is ready. Others monitor the web automatically. There's no central registry to submit to — each model's team handles discovery differently.

Your file doesn't prevent all AI use of your content. Models trained before your file existed will still contain your text. Models that don't check for llms.txt will ignore your rules. And models operating under fair use or similar legal doctrines may use your content regardless. What your file does is communicate your preferences clearly to the models that do respect it, and it creates a record of what you wanted.

Common mistakes to avoid

The most common mistake is uploading the file to the wrong location. It must be at the root of your domain (yoursite.com/llms.txt), not in a subfolder like yoursite.com/files/llms.txt. If it's in the wrong place, crawlers won't find it.

The second mistake is using the wrong file format. Save it as plain text only — .txt extension, no hidden formatting. If you copy and paste from a formatted document, invisible characters can break the file. Always save from a plain text editor.

The third mistake is being too restrictive without thinking through the consequences. If you disallow everything, AI models won't index your site at all, which means your content won't appear in their outputs — but it also means they can't misrepresent your work. If you allow everything, your content will be indexed, which increases visibility but gives you less control. Choose the balance that matches your actual goals.

Frequently Asked Questions

Do I need an llms.txt file if I already have a robots.txt?

No, you don't need one — but they serve different purposes. robots.txt controls search engines; llms.txt controls AI models. You can have both, one, or neither. If you want to manage how AI models use your content separately from how search engines index it, create an llms.txt file.

Will an llms.txt file stop AI models from using my content?

It will stop responsible models that check for it. Models that don't check for llms.txt, or that were trained before your file existed, may still use your content. The file is a signal of your preferences, not a legal barrier. For stronger protection, you need robots.txt rules, terms of service, or legal action.

What should I put in the disallow section?

Disallow anything you don't want indexed: admin panels, user accounts, draft posts, private pages, or sensitive information. Be specific — "Disallow: /admin/" blocks only that folder, while "Disallow: /" blocks everything. Most sites allow their public content and disallow only backend systems.

Can I change my llms.txt file after I create it?

Yes. Edit the file, save it, and upload the new version. Crawlers will pick up the changes on their next visit. There's no waiting period or approval process — the new rules take effect as soon as the file is live.

What licensing terms should I use?

That depends on your goals. If you want to restrict commercial use, set "Commercial: false". If you require attribution, set "Attribution: true". If you don't want derivatives, set "Derivatives: false". You can also use standard licenses like Creative Commons — just reference them in the file.