Free SEO & GEO Tool
Crawler Access Checker
Check which crawlers are allowed or blocked by robots.txt rules, then verify with live server checks using each bot's real User-Agent.
How this tool works
This tool fetches your site's robots.txt and checks 36 crawlers across search engines, AI bots, and other services. It parses User-Agent rules and Disallow paths, then verifies each bot with a live server request using the bot's real User-Agent string.
Explore more tools
Meta Tag Analyzer
Full meta tag audit for any URL.
llms.txt Generator
Create AI crawler guides.
Heading Checker
Analyze H1-H6 hierarchy.
FAQ
What is a robots.txt file?+
A text file at your site's root (example.com/robots.txt) that tells crawlers which pages they can or cannot access. It uses User-Agent, Allow, and Disallow directives to control crawling behavior per bot.
What does "rules" mean vs "BLOCKED"?+
"Rules" means the bot can crawl your site but certain paths are restricted (e.g. /admin/, /api/). This is normal and healthy. "BLOCKED" means Disallow: / — the bot cannot access any page at all.
Why does the server check show 403 for some bots?+
A 403 means the server actively rejects that bot regardless of robots.txt rules. This is typically done via firewall rules, CDN settings, or server-side bot detection. The bot cannot access your site even if robots.txt allows it.
What happens without a robots.txt?+
All crawlers assume they have permission to access every page. This tool shows all 36 bots as "allowed" in that case. You can still control access via server-side rules (which the live check reveals).
What is Crawl-Delay?+
A robots.txt directive that tells bots to wait a certain number of seconds between requests. It limits crawling speed to reduce server load. Google ignores Crawl-Delay — use Search Console's crawl rate setting instead.
AI crawler check on every page
Lumina shows robots.txt rules, X-Robots-Tag, and AI traffic sources — automatically, for free.
Add Lumina to Chrome — Free