Editor's note — September 26, 2026: This article has been corrected to distinguish the optional llms.txt proposal from crawler permissions and established search requirements. Earlier wording overstated its effect on visibility. The original publication date is preserved.
Two files with different jobs
Robots.txt communicates crawling preferences to automated clients that honor it. Llms.txt offers a curated guide to useful content for agents that choose to consult it. One expresses crawl rules; the other helps a reader find its way. Neither is a security barrier, and an llms.txt file does not guarantee indexing, citations or rankings.
That distinction matters when deciding where to spend effort. Fix inaccessible pages, unclear navigation and unreliable content first. Add an agent guide when it serves a useful retrieval or integration need, rather than treating another file as a visibility shortcut.
What llms.txt actually is (and is not)
Llms.txt is a Markdown file, usually at your domain root, containing a short description and organized links. The llms.txt proposal describes on-demand guidance for agents and can also cover a site's subdirectory. It is a proposed convention, not a permission system or a requirement imposed by every answer engine.
The idea solves a real problem. When a language model tries to read a modern web page, it wades through navigation, cookie banners, scripts, and markup before it reaches your actual words. Llms.txt offers a distilled index: a short description of your site, then a curated list of links to your most important pages, ideally pointing at markdown or plain-text versions stripped of the clutter.
A few things it is not:
- It is not an access-control file. It cannot block a request or grant usage rights. Robots.txt communicates crawling preferences; authentication and server controls protect private resources.
- It is not a search guarantee. Publishing it does not establish that a particular engine consumes it or gives it ranking weight.
- It is not a substitute for good structure. Public pages still need clear text, useful links and consistent metadata, whether or not an agent uses the guide.
Where it belongs on your roadmap
For Google's AI Overviews and AI Mode, the published advice is ordinary SEO: allow crawling, make content discoverable through links, provide useful text and keep structured data consistent with the page. Google explicitly says additional machine-readable files, AI text files and special schema are unnecessary. Meeting those requirements still does not guarantee inclusion. Google's AI features guidance
An optional guide can still be useful. Documentation readers may need a concise map to API references; a business assistant may need a reliable route to a company's services or biography. Judge that benefit by whether an agent can find and accurately use the intended information. Do not present successful retrieval as proof of a ranking improvement.
The Access Ledger: a five-part checklist
Keep a record of the separate decisions about crawling, discovery and content reuse. This checklist keeps their purposes clear.
Separate search from training. Consult each provider's documented crawler roles before setting rules. For example, OpenAI distinguishes OAI-SearchBot for search from GPTBot for potential training use; those preferences are independent. A training preference is not a promised search advantage. OpenAI crawler documentation
Keep useful pages reachable. Check that public pages and their required resources are crawlable. On large sites, manage unwanted parameter combinations deliberately; crawl budget is a separate concern from protecting private information.
Curate the guide. List core services, definitive guides and reference material with short descriptions. Link to a fuller archive when necessary, so the starting point stays manageable.
Keep alternate formats synchronized. Markdown or full-text feeds can help tools consume content. Generate them from the same source as the public page, retain attribution and citations, and link back to the canonical article. They are optional interfaces, not replacement search pages.
Verify, then re-verify. Confirm the files load and their links resolve. Check search tools and server logs for actual crawling. A successful request with a bot's user-agent name alone does not prove requests from that provider's real crawler infrastructure will succeed.
Repeat those checks after changes to hosting, navigation or publishing. A correct file cannot compensate for an authentication wall, a blocked network request or a broken destination.
The tension you have to resolve
Crawler policy is a business decision. Search discovery, user-requested retrieval and model training serve different purposes, and providers offer different controls. Evaluate them separately rather than treating every AI request as the same use.
My recommendation for a public consultancy site is to keep useful material easy to discover and measure the visits and enquiries it produces. Training permission deserves its own decision; allowing a training crawler is not evidence that future answers will remember or recommend the business.
A site can deliberately allow public crawling without an llms.txt file. Its absence does not make the site unreadable or establish an SEO defect.
The takeaway
Use robots.txt for voluntary crawling instructions and llms.txt, when useful, for a concise content guide. Protect private material with actual access controls. Keep the canonical website readable, accurately sourced and linked, then test any additional feed or guide against a real reader's needs. That is a useful implementation choice without inventing a new ranking requirement.
If you are trying to decide how open your site should be to answer engines, and you want that call made with your economics on the table rather than by default, the channel is open by introduction. Bring your robots.txt and your traffic, and we will set your terms on purpose.
