Robots.txt Generator
Generate robots.txt crawler rules for search engines and AI bots. Control Google, Bing, GPTBot, ClaudeBot, and more. Runs in your browser.
Robots.txt Generator
Runs entirely in your browser — no server calls, no tracking.
Fill in the fields and click Generate robots.txt.
🔒 Your data never leaves this tab. This tool has no backend.
About the Robots.txt Generator
A robots.txt file sits at the root of your domain and tells crawlers which pages they are and are not allowed to fetch. Every major search engine (Google, Bing) and most AI crawlers respect the robots exclusion protocol. The file is read on every crawl, so changes take effect within days of uploading a new version.
AI crawler options explained
Allow all AI bots — your content can appear in AI search results (Bing AI, Perplexity, ChatGPT search) and may be cited in AI answers. Good for brand visibility. Allow search, block training — permits AI search indexing but signals no permission for training data use. GPTBot is for training; OAI-SearchBot is for ChatGPT search — they are different. Block all AI bots — maximum restriction; your content will not appear in AI-powered search or citations.
What robots.txt cannot do
Robots.txt prevents crawling but does not prevent indexing. If another site links to a blocked URL, Google may still index it as a "known URL" with no content. To prevent indexing, use a noindex meta robots tag on the page itself. Also, robots.txt is publicly readable — do not list private URLs thinking that hides them; it does the opposite.
The Disallow directive uses prefix matching: Disallow: /admin/ blocks all paths under /admin/. Use a trailing slash to block a directory; without it, Disallow: /admin also blocks /administrator.html.
Frequently asked questions
Disallow for Googlebot. However, if a page is linked from other sites, Google may still index it as a known URL (without crawling the content). A disallowed page can still appear in search results if Google has seen it linked. To prevent indexing completely, use noindex in the page head or HTTP headers.Disallow: /draft-post.html. This only blocks that specific URL. Robots.txt uses prefix matching, so to block only one file you must use its full path without a trailing wildcard. For dynamic pages, a noindex meta tag on the page itself is more reliable.Sitemap: https://yourdomain.com/sitemap.xml at the end. This tells crawlers where to find your full URL list without relying on Search Console submission. Google reads it automatically when it first discovers your robots.txt.Need a full technical SEO setup?
ruxox builds web products with robots.txt, sitemaps, schema, and Core Web Vitals as part of the base build.