robots.txt Generator

Build a valid robots.txt from a list of paths, with a sitemap declaration and a warning before you block anything important.

SEO ToolsRuns in your browserFree, no sign-up

Options

Preset

Block everything is correct for staging and catastrophic on a live site.

User-agent

Use * for all crawlers, or name one such as Googlebot.

Sitemap URL

Crawl-delay (seconds)

Zero omits the directive. Google ignores it; Bing and Yandex honour it.

Paths to disallow (one per line)

robots.txt

Output appears here as you type.

Ctrl + Enter to run

About the robots.txt Generator

Builds a robots.txt with the user-agent groups, allow and disallow rules and sitemap reference you need. The file controls crawling, and the most consequential misunderstanding about it is that crawling and indexing are different things.

`Disallow` stops a crawler fetching a URL; it does not keep that URL out of search results. Worse, a blocked page can never be read, so a `noindex` tag on it is never seen. To keep a page out of the index, allow the crawl and serve `noindex` — using both means neither works.

Frequently asked questions

Does robots.txt stop a page appearing in Google?

Not reliably, and this is the most consequential misunderstanding about the file. Disallow stops a crawler *fetching* a URL; it does not stop that URL being indexed. If other pages link to it, Google can list it with no title and a "No information is available for this page" snippet. Worse, because it cannot fetch the page it never sees a noindex tag there. To keep a page out of search, allow crawling and serve noindex — the two directives are in tension and using both means neither works.

What is the difference between Disallow and noindex?

Disallow in robots.txt is about crawling: do not fetch this. noindex, in a meta tag or an X-Robots-Tag header, is about indexing: do not list this in results. Use Disallow to protect crawl budget on infinite URL spaces like faceted search. Use noindex to keep a real page out of search results. Never use both on the same URL.

Where must the file live?

Exactly at the root of the origin — https://example.com/robots.txt. It cannot live in a subdirectory, and it applies per origin including scheme, subdomain and port. So https://example.com and https://www.example.com need their own files, and a robots.txt on the apex does not govern the www host.

Are wildcards supported?

Google, Bing and most major crawlers support * for any run of characters and $ to anchor the end of a URL, so "Disallow: /*.pdf$" blocks PDF files. Neither is in the original 1994 specification, so a small or old crawler may ignore them. When rules conflict, the most specific match wins and a tie favours Allow — which is how "Disallow: /" plus "Allow: /blog/" leaves the blog crawlable.

Should I block AI crawlers here?

You can, by naming their user-agents (GPTBot, ClaudeBot, CCBot, Google-Extended and others) with Disallow rules. It works only for crawlers that choose to honour robots.txt — the file is a request, not an access control. Anything genuinely private needs authentication, not a line in a text file that publicly lists where your private paths are.