llms.txt and Controlling How AI Crawlers Use Your Site

Two files now decide whether machines can read you

For twenty years, one small text file at the root of your domain governed the relationship between your site and the machines that crawl it. Now there are two. Robots.txt still tells crawlers where they may and may not go. And llms.txt, a newer proposal, tries to hand large language models a curated map of what actually matters on your site, in the clean format they prefer. Between them, these two files increasingly decide whether an answer engine can read your content, understand it, and cite you when it composes an answer.

I have run large programs for fifteen years, and I have watched teams pour months into content that answer engines never surface, for a reason nobody in the room thought to check: the machine was either blocked, or it was drowning in your raw HTML and gave up. This is a technical decision with a business consequence, and most sites are making it by accident. Here is how to make it on purpose.

What llms.txt actually is (and is not)

Llms.txt is a plain markdown file you place at your domain root. It is a proposed convention, not a mandate anyone is required to honor, and that distinction matters. Think of it less as a lock and more as a concierge. Where robots.txt says "you may not enter this hallway," llms.txt says "here are the rooms worth your time, and here is the clean version of each."

The idea solves a real problem. When a language model tries to read a modern web page, it wades through navigation, cookie banners, scripts, and markup before it reaches your actual words. Llms.txt offers a distilled index: a short description of your site, then a curated list of links to your most important pages, ideally pointing at markdown or plain-text versions stripped of the clutter.

A few things it is not:

• It is not an access-control file. It does not block anything. If you want a crawler kept out, that job belongs to robots.txt and the discipline of crawl control, not here.
• It is not universally obeyed. Adoption among the major answer engines is uneven, and you should treat it as a signal you offer, not a rule you enforce.
• It is not a substitute for good structure. A clean site that machines can already parse benefits less from it than a heavy, script-driven one.

Why this belongs on your roadmap now

The reason to care is the same reason generative engine optimization has moved from a curiosity to a line item. Answer engines are now an interface between your content and a growing share of your audience, and they read differently than a person does. When the engine composes a response, it pulls from sources it could fetch cleanly and trust. If your best page is buried three redirects deep in a swamp of JavaScript, you are not in the running, no matter how good the writing is.

This is the quiet mechanics behind the click economy that answer engines have created. Visibility is shifting from "did I rank" to "was I readable and citable at the moment the answer was assembled." Llms.txt is one lever on the readable-and-citable side of that equation. It will not save weak content. It will make sure strong content is not lost in transit.

The Access Ledger: a five-part checklist

Do not treat these files as a set-and-forget afterthought. Treat access as a ledger you keep deliberately, the same way you would manage any other finite resource. Here is the checklist I walk through.

• Decide who gets in. In robots.txt, name the crawlers you welcome and the ones you do not. Answer-engine crawlers each announce themselves with a user agent. Choose deliberately whether you want your content used for answers, and record the decision so it is not silently reversed in a future deploy.

• Protect the swamp, not the storefront. The point of blocking is never to hide your good pages. It is to stop crawlers from wasting fetches on filter combinations, session URLs, and duplicates. This is the same waste that drains crawl budget on large sites, and the discipline transfers directly to AI crawlers, which are just as capable of getting lost.

• Curate, do not dump, in llms.txt. The value is in selection. List your genuinely important pages, the ones you would want quoted: your core service descriptions, your definitive guides, your reference material. A file that links to everything is a file that guides nothing.

• Serve clean versions. Where you can, point llms.txt entries at markdown or lightweight text renditions of each page. The less a machine has to fight your markup, the more faithfully it reproduces your meaning. Fidelity of citation is downstream of fidelity of parsing.

• Verify, then re-verify. Fetch both files as a crawler would. Confirm robots.txt is not accidentally blocking a section you meant to expose, and confirm every link in llms.txt resolves to a live, clean page. Then put it on a cadence, because a migration or a template change will break it when you are not looking.

That last point is where most failures live. I have seen a single stray line in robots.txt quietly wall off an entire content library, and nobody noticed until traffic told the story months later. Access files are load-bearing precisely because they are small and easy to ignore.

The tension you have to resolve

There is a genuine strategic question underneath all of this, and I will not pretend it is settled. Letting answer engines read and cite you can grow your reach and cost you the click at the same time. Blocking them protects the click and can erase you from the answer entirely. That is a real trade, and it depends on your model.

My honest read: for most businesses that earn trust by being found and understood, being present in the answer is worth more than the click you might have kept by hiding. Presence compounds. Absence is invisible, and you cannot measure the revenue from a mention you were never eligible for. But this is a decision to make with your own economics in front of you, not a rule to copy from anyone, including me.

What you should not do is make the decision by neglect. An empty robots.txt and no llms.txt is still a choice. It is just an unconsidered one.

The takeaway

Llms.txt and robots.txt are the two small files that now sit between your content and the machines composing answers for your audience. One controls access; the other curates attention. Neither is a magic switch, and neither is obeyed by everyone, but together they let you set the terms of the relationship instead of leaving them to chance. Curate your best pages, serve them clean, block only the swamp, and verify the whole thing on a schedule. The work is unglamorous and it takes an afternoon. The cost of skipping it is being unreadable at the exact moment you needed to be understood.

If you are trying to decide how open your site should be to answer engines, and you want that call made with your economics on the table rather than by default, the channel is open by introduction. Bring your robots.txt and your traffic, and we will set your terms on purpose.