Free SEO Tool · Updated 2026
Free Robots.txt Generator
Generate a correct, safe robots.txt file with your sitemap URL and any custom disallow rules, avoiding the mistakes that accidentally hide a whole site from Google.
Want your site actually built for SEO?
Every page we build for clients ships with clean metadata, schema and technical SEO baked in from day one.
Request a Free Site Survey WhatsApp an EngineerWhat robots.txt actually does
robots.txt is a plain-text file at the root of your domain that tells well-behaved crawlers which parts of the site they may or may not request. It is a courtesy instruction, not a security mechanism — it does not prevent a page from being indexed if it's linked from elsewhere, and it does not password-protect anything.
Common robots.txt mistakes that hurt rankings
- Accidentally blocking
/entirely, which tells every crawler to ignore the whole site. - Blocking CSS or JS folders, which can prevent Google from rendering the page correctly for mobile-friendliness checks.
- Forgetting the
Sitemap:line, which is a simple, free way to point crawlers straight to your full URL list. - Using
Disallowto try to remove a page from Google's index — this doesn't work reliably; use anoindexmeta tag or Search Console removal instead.
What's genuinely worth blocking
Typically only utility pages with no SEO value and duplicate-content risk: shopping cart, wishlist, internal search results pages, and admin/staging areas. Everything that could rank — services, articles, products — should stay crawlable.
Frequently asked questions
Not reliably. Disallowing a URL stops Google from crawling it, but if the URL is linked from elsewhere it can still be indexed with no description. Use a noindex meta tag to reliably keep a page out of search results.
If you don't have one, most crawlers assume the entire site is crawlable, which is fine for many small sites. A robots.txt file becomes useful once you have pages you genuinely want to keep away from crawlers, like a cart or admin area.
At the root of your domain, e.g. https://example.com/robots.txt — it will not be read from any subfolder.
Yes, it costs nothing and gives every crawler an immediate, explicit pointer to your full sitemap without waiting for it to be discovered another way.