Robots.txt Generator
Generate robots.txt crawler rules for search engines and AI bots. Control Google, Bing, GPTBot, ClaudeBot, and more. Runs in your browser.
Robots.txt Generator
Runs entirely in your browser — no server calls, no tracking.
Fill in the fields and click Generate robots.txt.
🔒 Your data never leaves this tab. This tool has no backend.
About the Robots.txt Generator
A robots.txt file sits at the root of your domain and tells crawlers which pages they are and are not allowed to fetch. Every major search engine (Google, Bing) and most AI crawlers respect the robots exclusion protocol. The file is read on every crawl, so changes take effect within days of uploading a new version.
AI crawler options explained
Allow all AI bots — your content can appear in AI search results (Bing AI, Perplexity, ChatGPT search) and may be cited in AI answers. Good for brand visibility. Allow search, block training — permits AI search indexing but signals no permission for training data use. GPTBot is for training; OAI-SearchBot is for ChatGPT search — they are different. Block all AI bots — maximum restriction; your content will not appear in AI-powered search or citations.
What robots.txt cannot do
Robots.txt prevents crawling but does not prevent indexing. If another site links to a blocked URL, Google may still index it as a "known URL" with no content. To prevent indexing, use a noindex meta robots tag on the page itself. Also, robots.txt is publicly readable — do not list private URLs thinking that hides them; it does the opposite.
The Disallow directive uses prefix matching: Disallow: /admin/ blocks all paths under /admin/. Use a trailing slash to block a directory; without it, Disallow: /admin also blocks /administrator.html.
What robots.txt can and cannot do
robots.txt sits at the root of your domain and tells compliant crawlers which paths to leave alone. Its most important limitation is the one people get wrong most often.
Disallow does not mean noindex. Blocking a URL stops crawlers fetching it — it does not stop them listing it. A blocked page linked from elsewhere can still appear in results, showing just the URL. Worse, because the crawler cannot read the page, it never sees a noindex tag you put there. To keep a page out of an index, allow crawling and use noindex; to keep it private, use authentication.
| Directive | Effect |
|---|---|
User-agent: * | Applies the rules below to all crawlers |
Disallow: /admin/ | Do not crawl anything under /admin/ |
Allow: /admin/help | Exception to a broader Disallow |
Sitemap: | Absolute URL of your sitemap |
The mistakes that cost traffic
Disallow: / left in place after launch blocks the entire site — the single most expensive one-character SEO mistake there is. Blocking CSS and JavaScript is the other: Google renders pages to evaluate them, so a blocked stylesheet makes your site look broken to the crawler and can affect how it is assessed.
It is public, and advisory
Anyone can read yourdomain.com/robots.txt, so listing /secret-admin-panel/ advertises exactly what you hoped to hide. And compliance is voluntary: reputable crawlers respect it, malicious scrapers ignore it entirely. It is a traffic-management tool, never a security control.
Frequently asked questions
No. Disallow stops crawlers fetching a page, not listing it — a blocked URL linked from elsewhere can still show in results as a bare URL. Worse, a crawler that cannot fetch the page never sees a noindex tag on it. To keep a page out of the index, allow crawling and use noindex instead.
Disallow, in robots.txt, controls crawling. Noindex, a meta tag or HTTP header on the page itself, controls indexing. They work at different stages, and using Disallow when you meant noindex is the most common robots.txt mistake because it prevents the noindex from ever being read.
No. Google renders pages to evaluate them, so blocking stylesheets and scripts makes your site appear broken to the crawler and can hurt how it is assessed. Allow all rendering resources.
Not at all. The file is publicly readable, so listing a path like /secret-admin-panel/ advertises it. Compliance is also voluntary — reputable crawlers honour it, malicious scrapers ignore it. Use authentication for anything that genuinely needs protecting.
At the root of the domain, exactly at yourdomain.com/robots.txt, in lower case. It does not work in a subdirectory, and each subdomain needs its own file — blog.example.com is not covered by the file on example.com.
Disallow for Googlebot. However, if a page is linked from other sites, Google may still index it as a known URL (without crawling the content). A disallowed page can still appear in search results if Google has seen it linked. To prevent indexing completely, use noindex in the page head or HTTP headers.Disallow: /draft-post.html. This only blocks that specific URL. Robots.txt uses prefix matching, so to block only one file you must use its full path without a trailing wildcard. For dynamic pages, a noindex meta tag on the page itself is more reliable.Sitemap: https://yourdomain.com/sitemap.xml at the end. This tells crawlers where to find your full URL list without relying on Search Console submission. Google reads it automatically when it first discovers your robots.txt.Need a full technical SEO setup?
ruxox builds web products with robots.txt, sitemaps, schema, and Core Web Vitals as part of the base build.