Robots.txt Generator
Build a correct robots.txt in seconds — allow or block crawlers, add your sitemap, and optionally block AI bots.
Runs entirely in your browser · nothing is uploaded · free & unlimited
Preset
robots.txt
User-agent: * Disallow:
How it works
- 1Pick a preset — allow every crawler, block them all, or choose Custom to set exact rules.
- 2In Custom mode, list the paths to Disallow (and any exceptions to Allow), set an optional crawl-delay and add your sitemap URL.
- 3Copy the generated file or download robots.txt and upload it to the root of your domain (e.g. example.com/robots.txt).
Frequently asked questions
More free tools
Sitemap Finder & Checker
Find and inspect every sitemap on any website — instantly.
Sitemap Validator
Validate an XML sitemap for errors, compliance and SEO issues.
CSV to Markdown
Turn CSV into a clean Markdown table — in your browser.
JSON to Markdown
Convert JSON into readable Markdown tables and lists.
Turn your website into an AI assistant
Docurest reads your content and answers your visitors' questions instantly — 24/7. Free to start.