Robots.txt and the Art of Crawl Control

Editor's note — September 26, 2026: The AI crawler section now distinguishes search, training and requested retrieval. References to llms.txt have been corrected: it is an optional content guide, not permission enforcement or a ranking requirement. The original publication date is preserved.

The most powerful file almost nobody reads carefully

A single wrong rule can keep compliant crawlers away from the pages you want discovered. Robots.txt is a plain text document at the root of a site that communicates which URLs a crawler may fetch. It is optional, but when present its rules deserve careful review.

A rushed migration or a copied staging configuration can disallow important production pages. Understanding what this file does, and just as importantly what it does not do, is basic hygiene for anyone responsible for a site's visibility.

What robots.txt actually controls (and what it does not)

Here is the distinction that trips up even experienced teams. Robots.txt controls crawling, not indexing. A Disallow rule tells a compliant crawler not to fetch a URL. It does not tell search engines to keep that URL out of their index.

Compliance is voluntary: robots.txt is not authentication or a security barrier. Use server-side access controls for private information. Google's documentation explicitly distinguishes crawl management from keeping a page out of search or protecting it from access. Google's introduction to robots.txt

This sounds like a technicality. It is not. If you disallow a page in robots.txt but other sites link to it, that URL can still show up in results, usually as an ugly listing with no description, because the crawler was never allowed to read the page and see the noindex tag you thought would handle it. You blocked the crawler from seeing the very instruction that would have removed it.

So keep these two jobs separate in your head:

• To request that compliant crawlers stop fetching, use robots.txt. This can help manage unwanted crawling, especially on large sites.
• To stop indexing, use a noindex meta tag or header, and crucially, leave the page crawlable so the crawler can actually read the directive.

Confusing the two creates a preventable failure: the page looks handled, but the crawler cannot read the instruction intended to remove it from search.

What you should disallow

Used well, robots.txt is a scalpel for steering crawler attention toward pages that earn and away from pages that drain. On large sites this is directly tied to how you manage crawl budget, because every fetch spent on junk is a fetch not spent on a page that converts.

Good candidates to disallow:

• Internal search results pages. Near infinite, low value, and a classic crawl trap.
• Faceted and filtered URL combinations that generate thousands of near duplicate pages. This is often the biggest single leak, and worth mapping carefully before you touch anything.
• Cart and checkout URLs whose public variants create unwanted crawling. Use authentication for account data; robots.txt cannot protect it.
• Public staging or tooling paths, only as supplementary crawl guidance. Anything private belongs behind authentication or network controls, not merely in a robots exclusion list.
• Endless calendar or pagination parameters that spiral out into nowhere.

The through line: block the swamp, not the garden.

What you must never disallow

This is where the real damage happens. A few rules I treat as non negotiable:

• Keep required CSS and JavaScript crawlable. Search engines that render a page need its supporting resources. Blocking them can prevent those engines from understanding the content and layout you intended.
• Never disallow a page you also want deindexed. As above, blocking the crawl blocks the noindex from ever being seen.
• Keep your sitemap and exclusions consistent. Submitting a URL for discovery while disallowing its crawl sends conflicting instructions. Your sitemap and your robots file should agree, which is one more reason your sitemap strategy and your crawl rules should be designed as one system, not two.
• Never copy blanket staging rules into an indexable production site. Disallow: / requests that compliant crawlers fetch none of the site's paths. Confirm that any blanket exclusion is deliberate, and remember it does not secure staging or remove previously indexed URLs.

A five point robots.txt audit you can run today

Start with these five checks on the live site:

1. Fetch the live file at yourdomain.com/robots.txt and read every line out loud. If you cannot explain why a rule exists, that is a flag.
2. Confirm CSS and JS are crawlable. Test a key template in a rendering tool and make sure nothing critical is blocked.
3. Cross check against your sitemap. No URL should be both submitted and disallowed. Reconcile every conflict.
4. Look for a stray Disallow: / or an overly broad rule that catches more than intended. Broad patterns are where accidents hide.
5. Verify what crawlers actually do by pulling server logs. Directives are theory; logs are truth, and what server logs reveal that analytics cannot will tell you whether crawlers are honoring your rules or wasting budget somewhere you never expected.

These checks establish whether the crawl instructions match the pages you want discovered. They do not replace an indexing or content-quality review.

How robots.txt now shapes AI crawler access

AI providers use different clients for search discovery, model training and user-requested retrieval. The names and rules matter; do not assume one instruction covers every product from a provider.

• OpenAI documents OAI-SearchBot for search and GPTBot for content that may be used in training. Those settings are independent. ChatGPT-User performs user-initiated actions, for which robots rules may not apply. OpenAI crawler documentation
• Anthropic distinguishes Claude-SearchBot, ClaudeBot and Claude-User for search, potential training and requested retrieval respectively, and documents robots.txt controls for them. Anthropic crawler documentation

Record the intended policy, then verify both the robots rules and the hosting or firewall response. A user-agent string can be imitated; tests using that name are useful diagnostics but do not establish that a request came from the real provider. Check provider documentation and verified traffic before changing firewall rules.

Llms.txt is an optional guide to content, not a way to enforce crawling permissions or dictate how fetched material is used. Google's AI features guidance says no additional AI text files or special schema are required for AI Overviews and AI Mode. Keep search eligibility, training preferences and convenience formats separate. Google's AI features guidance

The takeaway

Treat robots.txt as a deliberate set of crawling instructions. Keep important public pages reachable, reconcile exclusions with your sitemap, and record search and training preferences separately. Use authentication for private resources and verify real crawler traffic in logs. A correct configuration removes avoidable barriers; it does not guarantee indexing, visibility or citations.

If you have inherited a robots.txt file nobody fully understands, or you are weighing how to handle AI crawler access without giving away the store, the channel is open by introduction. Bring the file and your logs, and we will make sure it is opening the doors you want and closing the ones you do not.