Why robots.txt matters for AI
With the rise of AI search (ChatGPT, Perplexity, Claude), many sites now block AI crawlers via robots.txt. This tool shows exactly which bots are allowed or blocked, and whether your content can be indexed by AI systems.
The Lumina extension checks robots.txt, X-Robots-Tag headers, and meta robots on every page — plus it detects llms.txt and shows AI traffic in your Google Analytics dashboard.
More tools
Meta Tag Analyzer
Full meta tag audit for any URL.
llms.txt Generator
Create AI crawler guides.
Heading Checker
Analyze H1-H6 hierarchy.
FAQ
Which AI crawlers should I check?+
GPTBot (OpenAI), ClaudeBot/anthropic-ai (Anthropic), PerplexityBot, Google-Extended (Gemini), Bytespider (TikTok), CCBot (Common Crawl), and cohere-ai. This tool checks all of them.
Does blocking a crawler remove existing training data?+
No. robots.txt only prevents future crawling. It cannot remove data already used for training. For that, you'd need to contact the AI provider directly.
AI crawler check on every page
Lumina shows robots.txt rules, X-Robots-Tag, and AI traffic sources — automatically, for free.
Add Lumina to Chrome — Free