robots.txt Generator
Generate a robots.txt file for your website with custom rules.
Add one or more user-agent sections. Updates appear instantly.
Frequently Asked Questions
Will a Disallow rule here keep a page out of Google results?
Not by itself. Google's own documentation is clear that robots.txt is mainly for crawl control, and this tool only generates the file text; it does not turn a disallowed URL into noindex.
Can I block AI crawlers without blocking normal search engines?
Yes. The “Block AI crawlers” preset creates an allow-all group plus separate disallow groups for GPTBot, CCBot, ClaudeBot, and Bytespider.
What happens if I leave both Allow and Disallow empty in a group?
The generator inserts Allow: / for that user-agent group. That fallback is hardcoded so a blank section still produces a valid-looking rule block.
Can I add multiple user-agent groups and crawl delays?
Yes. You can add and remove groups, give each group its own agent, allow list, disallow list, and a Crawl-delay value. The crawl-delay field is numeric and limited to 0 through 999 in the UI.
Should I use the Host line this tool can generate?
Use it only if you know your crawler supports it. The tool will include a Host directive when you fill that field, but host support is not universal across crawlers.