Loading...
A free robots file generator that builds a custom, SEO-friendly robots.txt file in seconds. Ready-made templates for WordPress, Shopify & Blogger, sitemap support, crawl-delay control, and one-click blocking of AI crawlers like GPTBot & ClaudeBot.
Pick a template
Click a quick-start template at the top of the tool — WordPress, Shopify, Blogger or Block AI Bots — or leave it blank to build from scratch.
Enter your domain
On the Basic tab, type your domain (e.g. example.com). The tool auto-adds your Sitemap URL and uses it across the file.
Set your rules
Use the Basic and Advanced tabs to toggle folders to block, add custom Disallow / Allow paths and a crawl-delay.
Block AI bots (optional)
Open the AI Bots tab and toggle crawlers like GPTBot or ClaudeBot to keep AI scrapers out while Google stays allowed.
Copy or download
Check the live preview and the SEO validation panel, then hit Copy or Download to grab your robots.txt.
Upload to your root
Place the file in your site root so it loads at yourdomain.com/robots.txt — done.
Built for SEOs, bloggers and store owners who want technical SEO done right — without editing files by hand.
Stop GPTBot, ChatGPT, ClaudeBot, Google-Extended, CCBot & more from scraping your content for AI training — while keeping Google & Bing fully allowed.
Battle-tested robots.txt presets for WordPress, Shopify, Blogger and eCommerce sites — no syntax knowledge required.
Add one or many XML sitemaps so Google and Bing discover every page, post, product and image on your site.
Disallow filters, faceted navigation and parameter URLs to focus crawlers on the pages that actually rank.
Instant warnings catch mistakes — like accidentally blocking your whole site or hiding CSS and JavaScript from Google.
Jump straight to Google's robots.txt testing tool to validate your file before you deploy it.
Follow these six steps to generate and publish your robots.txt with the tool above.
Click a quick-start template at the top of the tool — WordPress, Shopify, Blogger or Block AI Bots — or leave it blank to build from scratch.
On the Basic tab, type your domain (e.g. example.com). The tool auto-adds your Sitemap URL and uses it across the file.
Use the Basic and Advanced tabs to toggle folders to block, add custom Disallow / Allow paths and a crawl-delay.
Open the AI Bots tab and toggle crawlers like GPTBot or ClaudeBot to keep AI scrapers out while Google stays allowed.
Check the live preview and the SEO validation panel, then hit Copy or Download to grab your robots.txt.
Place the file in your site root so it loads at yourdomain.com/robots.txt — done.
Copy-paste starting points for the most common setups. Generate your own customized version with the tool above.
User-agent: * Allow: / Sitemap: https://example.com/sitemap.xml
User-agent: * Disallow: /
User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: /
User-agent: * Allow: / Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Sitemap: https://example.com/sitemap_index.xml
| Directive | What it does |
|---|---|
| User-agent | Names the crawler a rule block applies to (* = all bots). |
| Disallow | Blocks crawlers from a path or pattern. |
| Allow | Permits a path, even inside a disallowed folder. |
| Sitemap | Points crawlers to your XML sitemap (absolute URL). |
| Crawl-delay | Seconds to wait between requests (Bing/Yandex). |
| Host | Declares the preferred domain (used by Yandex). |
| * and $ | Wildcard for any characters, $ matches the URL end. |
Robots.txt is a simple text file placed in the root of your website that follows the Robots Exclusion Protocol. It tells search engine crawlers — and now AI scrapers — which areas of your site they are allowed to request. A well-built robots.txt protects your crawl budget, keeps duplicate and low-value URLs out of crawling, and is one of the foundations of technical SEO.
This robots file generator produces a file Googlebot reads exactly the way Google documents it. Add your sitemap so Google discovers every URL, keep CSS and JavaScript allowed so Google can render your pages, and use Disallow only for the sections you truly want kept out of crawling. Remember that Googlebot ignores crawl-delay — set your crawl rate in Google Search Console instead — and that blocking a URL here stops crawling but does not remove an already-indexed page (use a noindex tag for that).
Whether you call it a robots file generator, a robot file generator or a robots.txt maker, the goal is the same: a clean, valid file at the root of your domain. Pick a WordPress, Shopify or Blogger template above, generate your file, and validate it with the built-in SEO checks before you upload it to yourdomain.com/robots.txt.
A robots.txt is a plain-text file in your site's root that tells search engine crawlers which URLs they may or may not request. It controls crawler access, protects your crawl budget, and keeps low-value or duplicate pages out of crawling.
Pick a template (WordPress, Shopify, Blogger or Custom) in the generator above, add your domain and sitemap, choose which folders or AI bots to block, then copy or download the file and upload it to your root directory at yourdomain.com/robots.txt.
Add a User-agent block for each AI crawler (GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, CCBot, PerplexityBot and more) followed by 'Disallow: /'. Use the 'Block AI Bots' template above to add all of them with one click while keeping Google and Bing allowed.
Upload it to the root of your domain so it is reachable at https://yourdomain.com/robots.txt. Search engines only read robots.txt from the root — it will not work inside a sub-folder.
No. Robots.txt blocks crawling but does not guarantee removal. A blocked URL can still appear in search results without a snippet. To remove a page from the index, use a noindex meta tag (and allow crawling) or password-protect it.
Googlebot ignores crawl-delay; set the crawl rate in Google Search Console instead. Bing, Yahoo and Yandex do honor crawl-delay, so it is still useful for those engines.
Without https:// or www — used to build your Sitemap URL.
Block all search engines
Use only for staging or private sites
Allow CSS / JS / image resources
Recommended — helps Google render pages
Disallow /admin/
Disallow /private/
Note: Googlebot ignores crawl-delay. Bing & Yandex respect it.