Robots.txt Generator: Crawl Rules That Do What You Meant
robots.txt is four lines of text that can quietly remove a site from search, and the rules that govern it are less obvious than they look. Paths are prefix matches rather than exact ones, the longest matching rule wins rather than the first, and blocking a page in robots.txt is not the same as keeping it out of the index. This generator builds the file with per crawler groups, sitemap lines and one click blocking of AI training bots, then tests any path against your rules and tells you which one applied and why. Everything runs in your browser.
Table of Contents
What robots.txt Actually Does
It asks crawlers not to fetch certain paths. That is the whole mechanism. It is a published request that well behaved crawlers honour voluntarily.
It is not access control. Anything ignoring the file simply fetches whatever it likes, and the file itself is public, so listing a secret directory tells the world exactly where to look.
It lives at the root and nowhere else. A crawler requests the file at the top of the domain. A copy in a subfolder is never read, and each subdomain needs its own.
Its real value is crawl efficiency. Keeping crawlers out of faceted filters, internal search and endless parameter combinations means the pages that matter get visited more often.
How to Use This Generator
Start from the preset closest to your site. The typical and WordPress presets cover the usual paths, and the staging preset blocks everything, which is the one case where that is correct.
Add a group when one crawler needs different treatment. A crawler uses the most specific group that names it and ignores every other group entirely, including the wildcard.
Test the paths you care about. Enter a real URL path and the tester names the winning rule. This is where prefix matching surprises people, and where the surprise is cheap to discover.
Read the warnings before you publish. They flag the specific patterns that most often remove a site from search by accident.
The Syntax in Full
User-agent names the crawler. An asterisk means every crawler that does not have a group of its own, and several User-agent lines can share one set of rules.
Disallow and Allow take a path. An empty Disallow means nothing is disallowed, and Disallow with a single slash blocks everything.
Two wildcards are widely supported. An asterisk matches any run of characters and a dollar sign anchors the rule to the end of the path, which is how you target a file extension.
Sitemap is independent of the groups. It is an absolute URL and applies to every crawler regardless of where it appears, so it conventionally goes at the end.
How Matching Really Works
Rules are prefix matches. Disallow: /admin blocks /admin, /admin/users and also /administrator, because the rule only has to match the start of the path. A trailing slash is what limits it to the directory.
The longest match wins, not the first. Order in the file does not decide anything. A narrow Allow beats a broad Disallow because it is longer, which is what makes the common pattern of blocking a directory and permitting one file inside it work.
Equal length is the only tie break. When a matching Allow and Disallow have exactly the same length, Allow wins. That is the only situation where the rule type decides.
One group applies, not several. A crawler picks the group naming it and ignores every other group, so a rule in the wildcard block does not apply to Googlebot if Googlebot has its own block.
Why It Cannot Hide a Page
Blocked is not the same as excluded. A page blocked in robots.txt can still appear in results if other sites link to it, usually with no description because the crawler was never allowed to read it.
Blocking defeats the tag that would work. A noindex meta tag can only be obeyed if the crawler is allowed to fetch the page and read it. Blocking it guarantees the tag is never seen.
To keep a page out of the index, allow the crawl. Let the crawler in and serve a noindex tag or header. To keep it private, put it behind authentication, since neither mechanism is a security control.
Blocking AI Crawlers
They are separate from search crawlers. Google-Extended controls AI training use and has no effect on Google Search, so blocking it costs you nothing in rankings.
Training and answering are different bots. One crawler gathers material for training, another fetches a page because a user asked about it right now. Blocking the second can remove you from AI answers that would have cited you.
Common Crawl feeds many datasets at once. CCBot is the source behind a great deal of training data, so it is often the most consequential single entry in the list.
Compliance remains voluntary. The major operators publish their user agents and state that they honour robots.txt. Nothing enforces it, so treat it as a stated preference.
What Is Worth Blocking
Internal search results. They generate unlimited low value URLs and are the classic crawl budget drain.
Faceted navigation and sort parameters. Every combination is a distinct URL with near identical content, and a shop can produce millions of them.
Cart, checkout and account pages. Useless in search, and often personalised or session dependent.
Not your CSS and JavaScript. Google renders pages, so blocking assets means it sees a broken version of your site. This is a genuinely damaging mistake and the warnings above flag it.
Sitemaps and Crawl Budget
One line advertises your sitemap to everyone. It works without any account or submission, and multiple sitemap lines are perfectly valid.
It must be an absolute URL. Unlike Disallow paths, the sitemap line takes a full URL including the scheme and host.
Crawl budget matters at scale. For a few hundred pages it is largely theoretical. For hundreds of thousands, stopping crawlers wandering through parameter combinations is one of the highest value things robots.txt does.
Common Mistakes to Avoid
Shipping the staging file to production. Disallow: / for the wildcard agent removes an entire site from search, and it is the single most common robots.txt incident.
Assuming Disallow: /* is narrower than Disallow: /. It is not. Both block everything, because the rule is already a prefix match.
Using it to hide sensitive paths. The file is public, so listing a private directory publishes its location to anyone curious enough to look.
Blocking a page you want deindexed. It has the opposite effect, because the noindex tag can never be read. Allow the crawl and use the tag.
Frequently Asked Questions
A reminder: robots.txt is a public file and a voluntary request. It controls crawling, not indexing and not access. Anything that must stay private belongs behind authentication, not behind a Disallow line.