Robots.txt Generator
Generate correct crawler directives with normalized paths and validated sitemap URLs. Includes presets for common scenarios.
Robots.txt Generator
Crawler rulesPresets
Separate multiple crawlers with commas or spaces. Use * for every crawler.
Allow and disallow rules
Paths must start with /. An empty Disallow path allows everything.
Seconds between requests. Not all crawlers honor this directive.
Must be an absolute http:// or https:// URL.
Used by some crawlers such as Yandex. Enter a hostname, not a URL.
About Robots.txt Generator
Build a valid robots.txt with user agents, allow and disallow rules, crawl delay, and sitemap.
This tool runs locally in your browser. It does not upload your input, require an account, or store your data on a server. If a feature needs a network request, the interface asks before sending anything.