Free · no signup · browser-local
Robots.txt Generator with Crawler Groups and Sitemap URLs
Create Allow and Disallow path groups for specific crawlers, add sitemap URLs and download robots.txt. Review the text before installing it on your own domain.
Crawler rules
Live output
Getting started
Add a user-agent group, enter one path per line, and export the text to your domain root.
Before you use the result
Robots rules guide compliant crawlers; they do not protect private content. This tool does not fetch or install the file.
Try an example
Allow /assets/ and disallow /draft/ for *, then include https://example.com/sitemap.xml.
How it works
Each group becomes explicit User-agent and path rules. Duplicate paths are removed and line-break injection is rejected.
Frequently asked questions
Build crawler groups with Allow, Disallow and sitemap declarations.
How do I use Robots.txt Generator?
Add a user-agent group, enter one path per line, and export the text to your domain root.
Does this tool upload my data?
Inputs and files are processed in your browser. Page-view analytics do not contain your tool input.
What should I check before using the result?
Robots rules guide compliant crawlers; they do not protect private content. This tool does not fetch or install the file.