Delimiter.live
Developer Tools Runs in your browser Free · no account

robots.txt Generator

The robots.txt generator produces a correctly formed file from the paths you want excluded and your sitemap URL, and can add a group blocking the major AI training and answer-engine crawlers. It writes each group contiguously, because a blank line inside a group can orphan its rules in stricter parsers.

Options
robots.txt

Set your options and press the generate button.

Results update automatically and are calculated on your device. Nothing you type is sent to a server.

Runs in your browser. Processing happens entirely in your browser. Nothing you enter is sent to Delimiter.live.

How to use the robots.txt Generator

  1. List the paths you want crawlers to skip, one per line — admin areas, internal endpoints, search result pages.
  2. Add your sitemap URL as an absolute address.
  3. Decide whether to block the AI crawlers; allowing them is the default.
  4. Generate, then upload the file to the root of your domain as /robots.txt.

Example

Input
Disallow /admin, sitemap URL
Output
User-agent: *
Allow: /
Disallow: /admin

Sitemap: https://example.com/sitemap.xml

Common use cases

  • Creating a robots.txt for a new site.
  • Keeping crawlers out of an admin area or a search results path.
  • Publishing a sitemap location so search engines find it without submission.
  • Opting out of AI training crawlers while staying open to search engines.

Limitations and things to watch for

  • robots.txt controls crawling, not indexing. A disallowed URL can still appear in search results if other sites link to it, because the crawler is told not to fetch it and therefore never sees a noindex tag. To keep a page out of results, allow crawling and use a noindex meta tag or header.
  • It is a public file and a request, not a control. Anyone can read it, and a badly behaved crawler can ignore it entirely. Never use it to hide sensitive paths — listing them tells everyone exactly where they are.
  • Crawl-delay is not supported by Google, which uses its own rate limiting and the Search Console setting instead. Bing and some others do honour it.
  • The file must sit at the root of the domain. A robots.txt in a subdirectory is ignored, and each subdomain needs its own.

Frequently asked questions

Does robots.txt keep a page out of Google?
No, and this is the most common misunderstanding. It stops the page being crawled, which means the crawler never reads any noindex instruction on it — so a blocked URL can still be listed from external links, showing a bare URL with no description. To exclude a page, let it be crawled and serve a noindex tag.
Should I block AI crawlers?
It depends what you want. GPTBot, CCBot and Google-Extended relate mostly to model training; OAI-SearchBot, ChatGPT-User and PerplexityBot fetch pages to cite them in answers, which can send you traffic. Blocking everything gives up the second to prevent the first.
Why does each blocked crawler group repeat the rules?
Because a crawler obeys only the most specific group matching it. Naming a crawler detaches it from the wildcard rules entirely, so any group that names crawlers must carry its own complete set.
Where does the file go?
At the root of the domain, exactly at /robots.txt. A file anywhere else is ignored, and every subdomain needs its own — a robots.txt on the apex does not cover blog.example.com.
Can I use wildcards?
Google and Bing support * for any sequence and $ for the end of a URL, which is how you exclude patterns such as /*?q=. They are extensions rather than part of the original standard, so less capable crawlers may ignore them.

Share this tool