Free Robots.txt Generator
Build a valid robots.txt from simple options, then toggle which AI crawlers like GPTBot and CCBot can train on your content. No signup.
- 100% free
- No signup
- Unlimited use
A robots.txt generator builds a valid robots.txt file from simple options: pick allow-all, block-all or WordPress defaults, add disallow paths and a sitemap line, and toggle which AI crawlers to block by name. This one covers GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and Bytespider, then outputs a ready-to-paste file you save at yoursite.com/robots.txt.
User-agent: * Disallow: /admin Disallow: /cart User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / Sitemap: https://example.com/sitemap.xml
Save this as robots.txt in your site root, at yoursite.com/robots.txt. robots.txt is advisory, so well-behaved crawlers obey it but it does not physically block anyone.
What a robots.txt file is and where it goes
robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may request. It lives at exactly one place, yoursite.com/robots.txt, and every well-behaved bot checks it before crawling. Each rule is one line of the form field: value. A group starts with a User-agent line naming the crawler, then one or more Disallow or Allow lines. Disallow lists a path prefix a crawler must not fetch, and Allow carves an exception back out of a Disallow. Field names are not case sensitive, but the path value is, so /Admin and /admin are treated as different. The file only controls crawling. It does not hide a page from being indexed if other sites link to it, so use a noindex tag or a password for anything truly private.
Allow all, block all, and the one character that flips the meaning
The two states people mix up look almost identical. An empty Disallow line, Disallow: with nothing after it, lets a crawler read your whole site. Disallow: / with a slash blocks it from the entire site. One character changes the meaning completely, so it is the most expensive typo in the file. The WordPress default sits in between: it blocks the /wp-admin/ folder but adds Allow: /wp-admin/admin-ajax.php so the scripts many themes and plugins depend on keep working. Start from the preset that matches your intent, then add specific Disallow paths like /cart or /search for pages that waste crawl budget without ranking.
Controlling whether AI trains on or cites your content
Blocking an AI crawler in robots.txt decides whether your content feeds AI assistants. The toggles above emit a named User-agent group for each bot you block. GPTBot is OpenAI's training crawler, ClaudeBot is Anthropic's, CCBot feeds the Common Crawl corpus that many models learn from, and Bytespider belongs to ByteDance. Google-Extended is a special case: it opts you out of Gemini training without touching Google Search, because Googlebot indexes your pages separately. The nuance that matters is that training and live citation are not the same thing. Blocking a training bot stops your words from being used to train a model, but it does not by itself remove you from the answers an assistant writes when it searches the web live. If your goal is to be cited by AI, you generally want to stay crawlable and publish clear, quotable content, which is what RankFirst does for you on autopilot. Reach for these blocks when you want your work kept out of training sets.
Common robots.txt mistakes to avoid
The costliest mistake is shipping Disallow: / on a live site, which quietly deindexes everything. Next is using robots.txt as a privacy tool: a blocked path can still show in search if another page links to it, and listing secret folders in a public file just advertises them. Do not block your CSS and JavaScript, because Google needs them to render and judge the page. Remember that Crawl-delay is ignored by Google, though Bing and Yahoo honor it, so set Google's crawl rate in Search Console instead. The file only governs the exact host it sits on, so a subdomain needs its own robots.txt. Test the result in Google Search Console before you rely on it.
Questions, answered
How do I create a robots.txt file?
Create a plain text file named robots.txt, add your rules, and upload it to the root of your domain so it loads at yoursite.com/robots.txt. Each rule is one line: a User-agent line names the crawler, then Disallow or Allow lines set the paths. This generator writes a valid file for you from the options above. Save it as UTF-8 and test it in Google Search Console.
How do I block GPTBot from my site?
Add a group that names the bot and disallows everything: User-agent: GPTBot on one line, then Disallow: / on the next. GPTBot is OpenAI's training crawler, so this keeps your pages out of future model training. Toggle GPTBot on in the AI crawlers section above and the tool writes that block for you. Blocking only affects future crawls, not data already collected.
Does blocking GPTBot stop ChatGPT from citing me?
Not on its own. GPTBot is OpenAI's training crawler, which is separate from the crawlers that fetch pages when ChatGPT searches the web to answer a question live. Blocking GPTBot keeps your content out of model training but does not by itself remove you from cited answers. If your goal is to be quoted by AI, stay crawlable and publish clear, self-contained content.
What is the difference between Disallow: / and an empty Disallow?
They are opposites that look nearly identical. Disallow: / with a slash blocks the crawler from your entire site. An empty Disallow, with nothing after the colon, explicitly allows the crawler to read everything. One character flips the meaning, so it is the easiest way to accidentally deindex a whole site. When you mean allow all, leave the path empty or omit the line.
Does Google support the crawl-delay directive?
No. Google has never supported Crawl-delay and ignores it entirely. Bing and Yahoo do honor it, treating the value as the minimum seconds a crawler should wait between requests. To slow Googlebot down, set the crawl rate in Google Search Console instead of adding Crawl-delay. This generator still includes the directive because the crawlers that respect it are worth targeting.
What is the WordPress default robots.txt?
WordPress serves a virtual robots.txt with three rules: User-agent: * , Disallow: /wp-admin/ , and Allow: /wp-admin/admin-ajax.php. It blocks the admin folder from crawlers while keeping admin-ajax.php reachable, since many themes and plugins call it on the front end. Pick the WordPress defaults preset above to start from exactly these rules. Uploading a real robots.txt file to your root overrides the virtual one.
More free tools
llms.txt Generator
Build a valid llms.txt file so AI assistants can find and cite your best pages. Free, no signup.
Open toolSERP Preview
See your title, URL and meta description as a live Google result, with pixel-accurate truncation for desktop and mobile. No signup.
Open toolURL Slug Generator
Turn any title into a clean, lowercase, SEO-friendly URL slug live as you type. Hyphens or underscores, optional stopword removal. No signup.
Open toolAI Answer Preview
See how ChatGPT or Perplexity would answer using your page, and how likely it is to be cited. Free, no signup.
Open toolThis tool fixes one job at a time. RankFirst publishes a fresh SEO & GEO article to your blog every single day.
We read your website, learn your voice and keywords, then write and publish for you on autopilot, so Google ranks you and AI assistants start naming your brand.