Delimiter.live
Comparison

noindex vs robots.txt

robots.txt controls crawling — whether a bot may fetch a URL. A noindex tag controls indexing — whether the page may appear in results. They are not interchangeable, and combining them backfires: a URL blocked in robots.txt is never fetched, so the noindex on it is never read, and the page can remain listed indefinitely.

robots.txt Generator

Build a valid robots.txt, with or without the AI crawlers.

Open the tool

This is the most consequential misunderstanding in technical SEO, because the intuitive fix makes the problem permanent. Someone finds a page they want out of Google, blocks it in robots.txt, and it stays in the index — often for years — as a bare URL with no title or description.

The reason is mechanical rather than mysterious. Google can list a URL it has never fetched, if other pages link to it. Blocking the fetch removes the only channel through which you could have told it not to.

What each one actually does

  • robots.txt Disallow: do not fetch this URL. Says nothing about indexing.
  • noindex meta tag or X-Robots-Tag header: you may fetch this, but do not list it.
  • The crucial asymmetry: a URL can be indexed without ever being crawled, from links alone.

Why combining them fails

To act on a noindex, a crawler has to read it, and to read it, it has to fetch the page. If robots.txt forbids the fetch, the noindex is invisible. The page stays in the index, usually showing just a URL with a note that no description is available — which is worse than either outcome you intended. If a page is already indexed and you want it gone, allow crawling, serve noindex, wait for it to be recrawled and dropped, and only then block it if you still want to.

When to use robots.txt

To save crawl budget on things that are worthless to fetch and that nobody links to.

  • Infinite or near-infinite URL spaces — faceted filters, calendars, search results.
  • Internal endpoints, health checks and asset paths with no standalone content.
  • Reducing load from an aggressive crawler on an expensive path.

When to use noindex

Whenever the goal is that the page does not appear in search results.

  • Thin or duplicate pages you still want reachable by visitors.
  • Staging or preview URLs that must stay accessible to the team.
  • Search result pages, paginated archives and tag pages of little standalone value.
  • Anything already indexed that you want removed.

Neither is a security measure

robots.txt is public and lists the paths you care about, so it advertises them. A noindex keeps a page out of results but leaves it fully reachable by anyone with the URL. If a page must not be seen, put it behind authentication — those are the only two tools here, and neither restricts access.

The canonical tag is a third thing

A canonical says "this content also lives at that URL, index that one instead". It is a consolidation hint for duplicates, not an exclusion. Use it for tracking-parameter variants and printer-friendly pages, where you want the ranking signals merged rather than the page suppressed.

Frequently asked questions

I blocked a page in robots.txt and it is still in Google. Why?
Because blocking prevents crawling, not indexing. Google can list a URL from links alone, and by blocking it you removed the only way to tell it otherwise. Unblock the URL, serve a noindex tag, and wait for it to be recrawled.
Can I use both on the same page?
Not usefully, and it is actively counterproductive. The robots.txt block stops the noindex being read. If you want both eventually, apply noindex first, confirm the page has dropped out, then add the block.
What is X-Robots-Tag?
The header equivalent of the noindex meta tag. It is how you exclude non-HTML responses — PDFs, images, JSON endpoints — which have no head section to put a meta tag in.
Does noindex waste crawl budget?
Slightly, since the page must still be fetched to be read. That is the trade, and for a handful of pages it is irrelevant. When the URL space is genuinely large, block it in robots.txt instead and accept that anything already indexed needs the noindex route first.
Where does the canonical tag fit?
It handles duplicates rather than exclusions. A canonical merges ranking signals onto one preferred URL; noindex removes a page from results entirely. Using a canonical when you meant noindex leaves the page eligible to rank.

Tools used in this guide

All tools →
All guides →

Last reviewed .