How-to guide

How to Write a robots.txt

A robots.txt tells crawlers which paths not to fetch. It lives at the root of the domain, groups rules under one or more User-agent lines, and can point to your sitemap. The crucial limit: it controls crawling, not indexing. A blocked URL can still be listed in search results, because the crawler never sees the page to read a noindex instruction.

robots.txt Generator

Build a valid robots.txt, with or without the AI crawlers.

Open the tool

The file is simple and its semantics are widely misunderstood in two specific ways, both of which cause real damage. The first is believing that blocking a path removes it from search. The second is not knowing that naming a crawler detaches it from the wildcard rules entirely.

Get those two right and the rest is a handful of lines.

Step by step

Put the file at the domain root
Exactly at /robots.txt. Anywhere else is ignored, and every subdomain needs its own — a file on the apex does not cover blog.example.com. It is served over both http and https and each is checked separately.
Understand how groups work
A group is one or more User-agent lines followed by rules. A crawler obeys only the single most specific group that matches it, so naming GPTBot means it ignores your wildcard group completely. Any group that names crawlers must carry its own full set of rules.
Keep each group contiguous
No blank line between a User-agent line and its rules. The original specification treats a blank line as a record separator, so a gap can orphan the rules from their group in stricter parsers. Comments are safe; blank lines separate groups.
Write the disallow rules
Paths are prefixes, so Disallow: /admin blocks /admin and everything under it. An empty Disallow means nothing is blocked. Google and Bing also support * for any sequence and $ for the end of a URL, which is how you exclude patterns such as /*?q=.
Add the sitemap line
Sitemap: followed by the absolute URL. It sits outside any group and applies to the whole file. It is the simplest way to make sure crawlers find your sitemap without submitting it anywhere.
Decide about AI crawlers deliberately
Training crawlers such as GPTBot, CCBot and Google-Extended are a different question from answer engines such as OAI-SearchBot and PerplexityBot, which fetch pages to cite them and can send traffic. Blocking everything gives up the second to prevent the first.
Do not use it to hide anything
The file is public and its whole purpose is to name paths. Listing /admin-secret tells every visitor exactly where it is. Protect sensitive paths with authentication; robots.txt is a request, not a control.

Example

Two groups. The named group repeats the rules because it would otherwise inherit nothing.

robots.txt

User-agent: GPTBot
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin
Disallow: /*?q=

Sitemap: https://example.com/sitemap.xml

What it does

GPTBot and CCBot: blocked entirely.
They read only their own group.

Everyone else: allowed, except
/admin and any URL with a q
parameter.

Sitemap applies to all crawlers.

Frequently asked questions

Will Disallow remove a page from Google?

No. It stops the page being fetched, which means any noindex tag on it is never read. A blocked URL can still be listed from external links, usually as a bare URL with no description. To remove a page, allow crawling and serve noindex.

Why must named groups repeat the rules?

Because a crawler applies only the most specific matching group. Adding User-agent: Googlebot with a single Allow line means Googlebot ignores every rule in your wildcard group, including exclusions you meant to apply to everyone.

Does Google support Crawl-delay?

No. Google ignores it and manages crawl rate itself, with a setting in Search Console. Bing and several others do honour it, so it is not useless — just not a way to slow Google down.

Should I block my search results pages?

Usually yes, because they are thin and effectively infinite, which wastes crawl budget. Combine it with a noindex tag on those pages so anything already indexed drops out — the Disallow alone will not remove them.

What happens if there is no robots.txt?

Everything is crawlable, which is a perfectly reasonable default for most sites. A 404 for the file is fine. What you should avoid is a server error, since some crawlers treat a 5xx as a temporary instruction to stop crawling altogether.

Tools used in this guide

All tools →

All guides →

Last reviewed 7 August 2026.