robots.txt Generator

Build a valid robots.txt with allow and disallow rules, sitemap lines and one-click blocking of AI crawlers. It runs in your browser and nothing is uploaded.

Choose your crawler rules

Which crawlers do the rules cover?

Allow paths (optional, one per line)

Disallow paths (one per line)

Start each path with a slash. Use * as a wildcard and $ for the end of a URL, for example /*.pdf$

Sitemap URLs (optional, one per line)

Crawl delay in seconds (0 = off)

Whole number from 0 to 60. Bing honors it, Google ignores it.

Your robots.txt

How to make a robots.txt file

A robots.txt file tells search and AI crawlers which parts of your site they may request. The preview updates as you type.

Which crawlers do the rules cover: all crawlers, block every crawler, WordPress defaults, block AI crawlers while keeping search engines, or one specific crawler such as Googlebot.
Allow paths: optional exceptions, for example /private/press-kit.html inside a blocked folder.
Disallow paths: one path per line, each starting with a slash. A * matches any characters and a $ marks the end of a URL.
Sitemap URLs: full addresses that start with https://, one per line.
Crawl delay in seconds: 0 to 60. Zero leaves the line out.
Press Copy or Download robots.txt and upload it to your domain root so it opens at https://yourdomain.com/robots.txt.

How the rules work

The file follows the Robots Exclusion Protocol, standardized as RFC 9309, as Google implements it. Rules sit in groups that begin with one or more User-agent lines. For a given URL, Google applies the most specific rule, meaning the one with the longest matching path, and if an Allow and a Disallow tie it uses the least restrictive one. Allow lines are written first so older crawlers that stop at the first match still honor your exceptions.

The WordPress choice writes Allow: /wp-admin/admin-ajax.php plus Disallow: /wp-admin/, /*?s= and /search/. It leaves /wp-includes/ open because Google needs the scripts and styles there.

Worked example

The page opens on All crawlers, disallow /admin/ and /private/, one sitemap line and a delay of 0. The output is 5 lines and 93 bytes:

User-agent: *
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

The page warns that example.com is a placeholder, so swap in your own domain. Choose Block AI crawlers and the file starts with 10 User-agent lines (GPTBot, Google-Extended, ClaudeBot and seven more) followed by Disallow: /, then the normal group for everyone else.

Limits and edge cases

robots.txt is a request, not a lock. Bad bots ignore it and the file is public, so never list secret URLs. Use a login or the .htaccess generator to restrict access.
Blocking stops crawling, not indexing. To keep a page out of results, leave it crawlable and add a noindex tag with the meta tag generator.
Every protocol, subdomain and port needs its own file. Google reads only the first 500 KiB.
Crawler names change, so recheck the AI list now and then. Pair this file with a sitemap so search engines find the pages you want crawled.

Frequently asked questions

Where do I put the robots.txt file?

Put it in the root folder of your site so it loads at https://yourdomain.com/robots.txt. A file in a subfolder is ignored. Each subdomain, such as blog.yourdomain.com, needs its own file.

How do I block ChatGPT and other AI crawlers in robots.txt?

Choose Block AI crawlers, keep search. The tool writes Disallow: / for GPTBot, ClaudeBot, Google-Extended, CCBot and the other ticked training crawlers and leaves Googlebot and Bingbot alone. Tick OAI-SearchBot too if you also want out of ChatGPT search results. ChatGPT-User acts when a person asks, and OpenAI says robots.txt rules may not apply to it.

Does robots.txt remove a page from Google?

No. It controls crawling, not indexing. A blocked URL can still be listed if other pages link to it, and Google cannot see a noindex tag on a page it is not allowed to crawl. To remove a page, let it be crawled and add a noindex tag, or delete it.

Does Google follow Crawl-delay in robots.txt?

No. Google lists user-agent, allow, disallow and sitemap as its supported fields and says other fields such as crawl-delay are not supported. Bing does honor Crawl-delay.

What happens when an Allow and a Disallow rule conflict?

Google applies the most specific rule, the one with the longest matching path. If they are equally specific it uses the least restrictive rule, which is Allow. So Disallow: /private/ with Allow: /private/press.html blocks the folder except that page.

Sources: Google Search Central, robots.txt specification; IETF RFC 9309; OpenAI crawler overview. Crawler lists checked October 2026.