🤖 Robots.txt Gen
Generate a robots.txt file free online. Control search engine crawling with correct syntax and sitemap declaration.
🤖 Robots.txt Gen
🔧 Loading tool…
About the Robots.txt Gen Tool
robots.txt is the first file a crawler requests from your domain, and it tells search engines which paths they may crawl. Used correctly it keeps admin pages, search result pages, and duplicate parameter URLs out of the crawl — focusing your crawl budget on pages that actually matter.
Used incorrectly it is one of the fastest ways to remove a site from Google entirely. A single misplaced Disallow: / blocks everything. This generator produces correct syntax with proper user-agent grouping and a sitemap declaration, so you get the intended behaviour rather than an accidental delisting.
How to Use Robots.txt Gen
- Choose which user agents the rules apply to, or select all crawlers.
- Add the paths you want to disallow, and any exceptions to allow.
- Add your sitemap URL so crawlers can find your full page list.
- Copy the output, save it as robots.txt, and upload it to your site root.
When to Use This Tool
- Blocking admin, login, and internal search pages from being crawled
- Declaring your sitemap location to search engines
- Preventing duplicate parameter URLs from consuming crawl budget
- Restricting specific crawlers while allowing others
- Setting up crawler rules for a new site launch
Things Worth Knowing
- robots.txt must be at the domain root, reachable at yoursite.com/robots.txt. Nowhere else works.
- Disallow prevents crawling, not indexing. Use a noindex meta tag to keep a page out of search results.
- The file is public and advisory. It is not a security measure — never use it to hide sensitive paths.
Quick Facts
| Tool name | Robots.txt Gen |
| Category | Developer Tools |
| Price | Free — no account, no trial, no watermark |
| Where it runs | Entirely in your browser (client-side) |
| Files uploaded | None — your data never leaves your device |
| File size limit | None imposed; limited only by device memory |
| Works offline | Yes, once the page has loaded |
| Platforms | Windows, macOS, Linux, Android, iOS |
Frequently Asked Questions
Does Disallow remove a page from Google?
No, and this is the most common misunderstanding. Disallow stops crawling, but a blocked URL can still be indexed if other sites link to it — appearing with no description. To remove a page from results, allow crawling and use a noindex meta tag.
Where exactly does the file go?
At the root of your domain, so it resolves at yoursite.com/robots.txt. Crawlers do not look anywhere else, and subdirectory copies are ignored.
Should I block my admin pages here?
You can, to save crawl budget, but remember robots.txt is publicly readable — listing /secret-admin/ tells everyone exactly where it is. Use authentication for actual security.
Do I need to declare my sitemap?
It is not required but strongly recommended. A Sitemap: line gives crawlers a direct route to your full URL list, which helps discovery on larger sites.