Meta robots tag

The meta robots tag is an HTML directive, placed in a page’s head, that controls how search engines treat that specific page — most commonly whether to index it (index/noindex) and whether to follow its links (follow/nofollow). It is the precise, page-level counterpart to the site-wide robots.txt file, and it is the correct tool for keeping a crawlable page out of the index.

How it works

Unlike robots.txt, which controls crawling, the meta robots tag (or its HTTP-header equivalent, X-Robots-Tag) controls indexing — and for it to work, the page must be crawlable, so the crawler can read the directive. A common mistake is blocking a URL in robots.txt and also adding noindex: the crawler never fetches the page, never sees the noindex, and the URL can still appear in results. Use noindex for pages that must exist for users but shouldn’t rank (thin faceted variants, internal search results, thank-you pages). The X-Robots-Tag header extends the same control to non-HTML files like PDFs.

Where it applies

• Ecommerce — noindex on low-value faceted and filter URLs that should stay usable but unindexed.
• Any site — internal search results, staging pages, and utility pages that shouldn’t rank.
• Enterprise — PDFs and documents controlled via the X-Robots-Tag header.

What matters from each seat

• In-house — decide index vs noindex per template deliberately; leaving it to defaults causes index bloat.
• Engineering — never noindex a page you also block in robots.txt; the directive won’t be seen.
• Agency and consulting — auditing accidental noindex tags is a classic recovery of pages that silently fell out of the index.