How to Create a Robots.txt File for SEO (Free Generator + Examples, 2026)
Learn how to create a robots.txt file for SEO the easy way. Free step-by-step guide with examples for WordPress, Shopify & Blogger — plus how to block AI bots like GPTBot. Generate yours in seconds with ShopYor.
Shayan Attique
Here's a question that trips up even experienced site owners: one tiny text file can quietly decide how Google sees your entire website. Get it right, and search engines crawl your best pages efficiently. Get it wrong, and you can accidentally hide your whole site from Google — it happens more often than you'd think.
That file is called robots.txt, and the good news is you don't need to be a developer to create one. In this guide I'll walk you through exactly what it does, how to write it, and the safest way to build a perfect one in under a minute using a free robots.txt generator — with real examples for WordPress, Shopify, and Blogger along the way.
Table of Contents
- What Is a Robots.txt File (and Why It Matters for SEO)
- How Robots.txt Actually Works
- The Easiest Way: Use a Free Robots.txt Generator
- Robots.txt Syntax: The Rules Explained
- Robots.txt Examples for WordPress, Shopify & Blogger
- How to Block AI Bots Like GPTBot and ChatGPT
- 5 Robots.txt Mistakes That Hurt Your SEO
- How to Upload and Test Your Robots.txt
- Frequently Asked Questions
- Conclusion
What Is a Robots.txt File (and Why It Matters for SEO)
A robots.txt file is a simple text document that sits in the root of your website — reachable at yourdomain.com/robots.txt. Its job is to give instructions to web crawlers (also called bots or spiders) about which parts of your site they're allowed to access.
When Googlebot arrives at your site, the very first thing it looks for is your robots.txt file. It reads your rules, then crawls accordingly. That makes this little file one of the most important — and most overlooked — pieces of technical SEO.
Why You Should Care About Robots.txt
A well-built robots.txt does three things that directly affect your rankings:
- Protects your crawl budget. Search engines only spend so much time crawling your site. Blocking low-value URLs (filters, search results, admin pages) means more crawl time for the pages you actually want ranked.
- Points crawlers to your sitemap. Adding your XML sitemap to robots.txt is the fastest way to help Google discover every page on your site.
- Keeps junk out of crawling. Duplicate content, thank-you pages, and parameter URLs can dilute your SEO. Robots.txt keeps crawlers focused.
For the official word straight from the source, Google's own robots.txt documentation is worth a bookmark.
How Robots.txt Actually Works
Robots.txt follows a standard called the Robots Exclusion Protocol. Every rule is built from two basic parts: a user-agent (which bot the rule applies to) and one or more directives (what that bot can or can't do).
Here's the simplest possible example:
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlIn plain English, that says: "Every bot (*) is allowed to crawl everything, and here's my sitemap." That's a perfectly valid, SEO-friendly robots.txt for most websites.
One important thing to understand: robots.txt controls crawling, not indexing. Blocking a page stops bots from reading it, but if other sites link to it, the URL can still show up in search results (just without a description). To truly remove a page, you need a noindex tag — not robots.txt.
The Easiest Way: Use a Free Robots.txt Generator
You can write robots.txt by hand in a text editor. But one stray slash can accidentally block your entire site — so unless you enjoy living dangerously, the smarter move is to use a tool that gets the syntax right for you.
Our free Robots.txt Generator does exactly that. Here's how to create your file in about 60 seconds:
- Pick a template. Choose a quick-start preset — WordPress, Shopify, Blogger, or "Block AI Bots" — or start from scratch.
- Enter your domain. Type in your website address and the tool automatically adds your sitemap URL.
- Set your rules. Toggle which folders to block, add custom Allow/Disallow paths, and set a crawl-delay if needed.
- Block AI bots (optional). Flip on crawlers like GPTBot or ClaudeBot to keep AI scrapers out while Google stays allowed.
- Copy or download. Check the live preview and the built-in SEO validation, then grab your file.
Why use the generator instead of copy-pasting? It includes a live SEO checker that warns you the moment you do something risky — like blocking your whole site or hiding CSS and JavaScript from Google. That safety net alone is worth it.
→ Create your free robots.txt file now
Robots.txt Syntax: The Rules Explained
Even if you use a generator, it helps to understand what each line means. Here are the directives you'll actually use:
| Directive | What it does |
|---|---|
| User-agent | Names the crawler a rule applies to. * means all bots. |
| Disallow | Blocks a path or pattern from being crawled. |
| Allow | Permits a path — even inside a disallowed folder. |
| Sitemap | Points crawlers to your XML sitemap (use the full URL). |
| Crawl-delay | Seconds to wait between requests (Bing & Yandex honor it; Googlebot ignores it). |
Two special characters give you fine control:
*— a wildcard that matches any sequence of characters. Example:Disallow: /*?blocks every URL containing a question mark.$— matches the end of a URL. Example:Disallow: /*.pdf$blocks all PDF files.
Robots.txt Examples for WordPress, Shopify & Blogger
Different platforms create different "junk" URLs, so the ideal robots.txt varies. Here are tested starting points you can generate instantly with the tool.
Robots.txt for WordPress
User-agent: *
Allow: /
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Sitemap: https://example.com/sitemap_index.xmlThis blocks the WordPress admin area and internal search results while keeping the AJAX endpoint (which themes need) accessible.
Robots.txt for Shopify / eCommerce
User-agent: *
Allow: /
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /*?*sort_by*
Disallow: /*?*filter*
Sitemap: https://example.com/sitemap.xmlOnline stores generate endless filter and sort URLs. Blocking them protects your crawl budget and prevents duplicate-content issues.
Robots.txt for Blogger
User-agent: *
Allow: /
Disallow: /search
Sitemap: https://example.com/sitemap.xmlBlogger's /search paths create thin, duplicate pages — blocking them keeps your blog's crawl focused on real posts.
Don't want to copy these manually? The generator builds the correct version for your exact domain in one click.
How to Block AI Bots Like GPTBot and ChatGPT
This is the big 2026 question: do you want AI companies training their models on your content for free? If the answer is no, robots.txt is your first line of defense.
AI crawlers each have their own user-agent name. To block them, you add a rule for each one:
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: PerplexityBot
Disallow: /The beauty of this approach is that it's surgical: you block the AI scrapers while leaving Googlebot and Bingbot completely free to crawl and rank your site. You lose nothing in search visibility.
Typing all those out is tedious, so the ShopYor generator has a one-click "Block AI Bots" template that adds GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more — instantly. Try it here.
5 Robots.txt Mistakes That Hurt Your SEO
I've audited hundreds of sites, and the same robots.txt errors come up again and again. Avoid these:
- Blocking your entire site by accident. A lone
Disallow: /underUser-agent: *hides everything from Google. This is the #1 traffic-killer — usually left over from a staging site. - Blocking CSS and JavaScript. Google needs to render your pages like a browser does. If you block your
/assets/or theme files, your pages may look broken to Google and rank worse. - Using robots.txt to hide private data. Robots.txt is public — anyone can read
yourdomain.com/robots.txt. Listing secret folders there is basically a map for snoopers. Use passwords instead. - Forgetting the sitemap line. Leaving out your sitemap means missing the easiest free SEO win available.
- Relying on crawl-delay for Google. Googlebot ignores
Crawl-delay. Control Google's crawl rate in Search Console instead.
The generator's live validation catches mistakes 1, 2 and 4 automatically — another reason to let a tool handle it.
How to Upload and Test Your Robots.txt
Once you've generated your file, here's how to put it live:
Step 1: Upload to Your Root Directory
- WordPress: Use an SEO plugin like Yoast or Rank Math (both have a robots.txt editor), or upload via your hosting file manager.
- Static / custom site: Drop
robots.txtinto your public root folder via FTP or your host's file manager. - Shopify: Edit the
robots.txt.liquidtemplate under your theme's code.
Step 2: Confirm It's Live
Open https://yourdomain.com/robots.txt in your browser. You should see your file exactly as you created it.
Step 3: Test It in Google Search Console
Use Google's robots.txt report in Search Console to make sure Google can read your file and that you haven't accidentally blocked anything important. The generator even includes a one-click link to Google's tester.
Bonus: While you're optimizing technical SEO, generate clean meta tags too with our free Meta Tag Generator, and explore the full collection of free SEO tools.
Frequently Asked Questions
What is a robots.txt file in simple terms?
A robots.txt is a small plain-text file that lives in the root of your website (yourdomain.com/robots.txt). It tells search engine crawlers like Googlebot which pages and folders they're allowed to visit and which to skip — like a doorman for the bots that crawl your site.
How do I create a robots.txt file for free?
The fastest way is to use a free robots.txt generator. Pick a template, add your domain and sitemap, toggle the folders or AI bots you want to block, then copy or download the file — no coding needed. Upload it to your site root and you're done.
Where do I put the robots.txt file on my website?
It must go in the root directory so it loads at https://yourdomain.com/robots.txt. Search engines only read robots.txt from the root — placing it in a sub-folder won't work. On WordPress you can add it via an SEO plugin; on most hosts you upload it through your file manager or FTP.
Can robots.txt block AI bots like ChatGPT and Claude?
Yes. Add a User-agent block for each AI crawler — GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, CCBot, PerplexityBot and others — followed by "Disallow: /". This blocks them from scraping your content for AI training while keeping Google and Bing fully allowed.
Will robots.txt remove my page from Google search?
No. Robots.txt only controls crawling, not indexing. A blocked URL can still appear in search results without a description. To fully remove a page, allow crawling and add a "noindex" meta tag, or password-protect the page instead.
Does every website need a robots.txt file?
Technically no — without a file, crawlers assume they can crawl everything. But it's strongly recommended, because it lets you point search engines to your sitemap, protect your crawl budget, and keep low-value URLs out of crawling. It takes one minute to create.
Conclusion
Your robots.txt file is small, but its impact on SEO is anything but. Done right, it guides search engines to your best content, protects your crawl budget, points bots to your sitemap, and now even keeps AI scrapers from harvesting your work.
Here's what you learned:
- What robots.txt is and why it's a foundation of technical SEO.
- The exact syntax — user-agents, Allow, Disallow, Sitemap and wildcards.
- Ready-to-use examples for WordPress, Shopify and Blogger.
- How to block AI bots like GPTBot and ClaudeBot in one move.
- The five mistakes that quietly tank your rankings — and how to avoid them.
You don't have to write a single line by hand. Build a perfect, SEO-ready robots.txt in under a minute with our free tool — complete with templates, AI-bot blocking, and live validation.
Written by
Shayan Attique
Sharing tips, tutorials & guides on the Shopyor blog.