Robots.txt

Robots.txt is a plain-text file at the root of a domain (for example, example.com/robots.txt) that tells crawlers which URL paths they may or may not request. It controls crawling, not indexing: a well-behaved bot obeys its Disallow rules, but a blocked URL can still appear in search results if other pages link to it. It is the first file most crawlers fetch on a site.

How it works

The file uses simple directives grouped by User-agent. Disallow lists path prefixes a bot should not crawl; Allow carves out exceptions; and a Sitemap line points to the XML sitemap. Rules are advisory — reputable search and AI crawlers honor them, malicious scrapers may not. A critical distinction: to keep a page out of the index, you need a noindex directive or authentication, not a Disallow, because a disallowed URL is never crawled and so its noindex is never seen. Mistakes here are high-blast-radius; a stray Disallow: / can deindex an entire site. Always validate changes before deploying.

Where it applies

Robots.txt matters most on large or resource-constrained sites.

• Ecommerce — blocking cart, checkout, and internal search URLs keeps crawlers focused on indexable inventory.
• Enterprise — staging, admin, and faceted paths are commonly disallowed to protect crawl budget.
• Marketplaces — controlling crawler access to low-value parameter URLs prevents wasted fetches at scale.
• Publishing — increasingly used to allow or disallow specific AI training and answer-engine crawlers.

What matters from each seat

• In-house — review robots.txt after every platform change; it is small, easy to break, and rarely tested.
• Agency and consulting — an early robots.txt check catches accidental site-wide blocks that silently tank traffic.
• Engineering — never let a staging Disallow: / ship to production; gate the file by environment and validate in CI.