AI Crawler & llms.txt Checker
Check how AI crawlers see your site — your llms.txt and robots.txt rules — and generate an llms.txt draft from your sitemap.
You've used all free runs.
How it works
- 1Enter your domain. We look for /llms.txt, read the links inside it, and check each one for broken or redirected URLs.
- 2We read your robots.txt and evaluate the rules for common AI crawlers, flagging blocks and conflicting allow/disallow rules.
- 3We use your sitemap to draft an editable llms.txt — grouped into documentation, products, support and policies — for you to refine.
Frequently asked questions
More free tools
Sitemap Finder & Checker
Find and inspect every sitemap on any website — instantly.
Sitemap Validator
Validate an XML sitemap for errors, compliance and SEO issues.
CSV to Markdown
Turn CSV into a clean Markdown table — in your browser.
JSON to Markdown
Convert JSON into readable Markdown tables and lists.
HTML to Markdown
Paste HTML and get clean Markdown — links, lists, tables and more.
Chatbot ROI Calculator
Estimate the support savings from an AI assistant.
Robots.txt Checker
Analyze any site's robots.txt — rules, sitemaps and issues.
Meta Tag & Open Graph Checker
Inspect a page's SEO, Open Graph and Twitter-card tags.
Sitemap URL Extractor
Pull every URL out of an XML sitemap, ready to copy.
AI FAQ Generator
Turn any URL or text into ready-to-publish FAQs with AI.
Ask Any Webpage
Ask a question and get an AI answer from any page.
XML to Markdown
Turn XML into a readable nested Markdown outline.
Webpage to Markdown
Convert any web page (by URL) into clean Markdown.
Website Word Counter
Count words, characters & reading time on any page.
Sitemap to Robots.txt
Generate a robots.txt — with sitemap and AI-crawler blocking.
AI Blog Title Generator
Get 10 catchy, SEO-friendly blog titles from a topic.
Sitemap Comparison
Diff two sitemaps — see URLs added and removed.
Sitemap Analytics
Metrics & insights from any XML sitemap.
Page Link Extractor
Get every link on a page, internal vs external.
Sitemap Index Generator
Turn a list of sitemaps into a sitemap index.
AI Meta Description Generator
3 SEO meta descriptions from a URL or text.
HTTP Header Checker
Inspect any URL's response headers, caching & security.
Redirect Checker
Trace a URL's full redirect chain, hop by hop.
Keyword Density Analyzer
Top words & phrases on a page, with density %.
UTM Campaign URL Builder
Build tagged campaign URLs for Google Analytics.
Slug Generator
Turn titles into clean, SEO-friendly URL slugs.
Meta Tag Generator
Generate SEO, Open Graph & Twitter meta tags.
Robots.txt Generator
Build a robots.txt with rules, sitemap & AI-crawler blocking.
JSON Formatter & Validator
Beautify, minify and validate JSON in your browser.
Base64 Encoder / Decoder
Encode and decode Base64 text (UTF-8 safe).
URL Encoder / Decoder
Percent-encode or decode URLs and query strings.
Text Case Converter
Convert text between UPPER, lower, Title, camelCase & more.
Word & Character Counter
Count words, characters, sentences & reading time.
Lorem Ipsum Generator
Generate placeholder text — paragraphs, sentences or words.
Password Generator
Create strong, random passwords (crypto-secure).
UUID Generator
Generate random UUIDs (v4) in bulk.
Google SERP Snippet Preview
Preview your title, URL & description as they may appear in Google.
Hash Generator (SHA)
SHA-1/256/384/512 hashes of any text.
JWT Decoder
Decode a JSON Web Token's header & payload.
Text Diff Checker
Compare two texts — see additions & removals.
Unix Timestamp Converter
Convert Unix timestamps to dates & back.
Color Converter
Convert between HEX, RGB & HSL.
Turn your website into an AI assistant
Docurest reads your content and answers your visitors' questions instantly — 24/7. Free to start.