AI Crawlers: What They Do and How to Control Them in 2026

An AI crawler is a bot that fetches web pages so a large language model can do something with them. In 2026 that something is one of three jobs: train the next model, populate a search index, or answer a specific user's question live. Each job has its own bot. The gap between "allow everything" and "block everything" is the part most robots.txt guides skip, and it's where the interesting decisions live.

Below: who the crawlers are, how much traffic they generate, and what 10 top news sites block, from an audit re-run on September 17, 2026. Lumina's Crawler Access Checker runs the same analysis against any URL.

What is an AI crawler?

On the wire, an AI crawler looks identical to a classic web crawler: same HTTP requests, same user-agent headers, same respect (usually) for robots.txt. What changes is what happens to the content after it's fetched.

Classic crawlers like Googlebot exist to rank pages against each other and return links in a search result. Users click through. AI crawlers skip the click. ChatGPT reads your page, reasons about the content, and writes an answer that may cite you or may not. That's the shift. Your content stops being a destination and becomes a source.

Because "AI crawler" is a job description rather than a strict category, the list evolves fast. The bots from OpenAI, Anthropic, Google, Perplexity, Apple, Meta, and Common Crawl are the ones that matter in 2026.

The major AI crawlers in 2026

The ones you'll see in your server logs:

Googlebot itself isn't on this list. It still fetches your pages for Search, and Google-Extended only controls what Google may do with them afterwards: train Gemini models and ground answers in Gemini Apps and Vertex AI. Google says Google-Extended has no effect on inclusion or ranking in Google Search.

Training vs. search: the split that matters

The single biggest 2024 to 2026 shift: OpenAI, Anthropic, and Perplexity all split their one bot into multiple bots with distinct jobs.

OpenAI started with GPTBot and ChatGPT-User in 2023 and added OAI-SearchBot in 2024. Anthropic runs Claude-SearchBot and Claude-User next to ClaudeBot. Perplexity uses PerplexityBot for indexing and Perplexity-User for live retrieval. The pattern is the same everywhere: one bot for training, one for indexing, one for live user-triggered fetches.

OpenAI extended the pattern again in April 2026 with OAI-AdsBot, a fourth bot for ChatGPT ads validation. It only visits pages advertisers submit as ad landing pages, checks them against ad policies, and decides when to show the ad. If you don't run ChatGPT ads, it never visits. If you do, blocking it means your ads can't be validated and won't serve.

If you block GPTBot only, you've stopped OpenAI from training on your content, but ChatGPT can still cite you when a user asks it a question, because ChatGPT-User is a different bot. Most publishers want that middle ground: no training, yes citation.

The catch: bots that fetch a page because a user asked for it mostly don't follow robots.txt. OpenAI changed its documentation for ChatGPT-User in December 2025 to say "robots.txt rules may not apply". Perplexity says Perplexity-User generally ignores robots.txt, and Google says the same about Google-Agent. Anthropic is the exception: its help center says all three of its bots, Claude-User included, honor robots.txt. So the split still works for training and indexing. For live user fetches, a robots.txt block is closer to a wish.

Concrete example. The Guardian lets GPTBot, OAI-SearchBot and ChatGPT-User through the * fallback and blocks ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot and CCBot by name. The NYT takes the opposite approach: after its 2023 lawsuit against OpenAI, it blocks every AI bot in the audit below. Wikipedia allows everything, which fits a CC BY-SA license that already permits re-use.

How much traffic AI crawlers generate

Cloudflare counted more than 50 billion AI crawler requests a day on its network in March 2025, just under 1% of all requests it sees. Most of that is training, not answering user queries. In Cloudflare's breakdown for July 2025, training made up nearly 80% of AI bot traffic, and in early August GPTBot and ClaudeBot together accounted for about half of all observed AI crawling.

Classic search still dwarfs the AI side. Vercel's edge logs for late 2024 put Googlebot at 4.5 billion requests per month versus GPTBot at 569 million, Claude at 370 million, AppleBot at 314 million, and PerplexityBot at 24 million. That's roughly an 8× gap to GPTBot and nearly 200× to PerplexityBot. AI crawlers are growing fast but haven't caught traditional search in raw volume, and probably won't for years.

Not every crawler follows the rules, either. Per Vercel's own data the major declared bots (GPTBot, Claude, AppleBot) respect robots.txt. The problem is the undeclared ones. A Wired investigation in June 2024 caught Perplexity bypassing robots.txt via an unnamed AWS-hosted crawler; Cloudflare publicly called out the same pattern in August 2025. If a crawler has decided to scrape you, robots.txt won't stop it.

Live audit: what 10 top sites actually block

On September 17, 2026, I pulled robots.txt from 10 high-traffic sites and evaluated it for 12 AI bots the way RFC 9309 prescribes: a bot's own user-agent group wins, the * group is the fallback, and a bot counts as blocked when the root path / is disallowed. Training bots get blocked far more often than retrieval bots. The bigger surprise is how much depends on which company runs the bot.

The most telling files are the ones that pick companies. WSJ's robots.txt blocks every bot through * and then opens a group for ChatGPT-User, GPTBot and OAI-SearchBot; its owner News Corp signed a content deal with OpenAI in 2024. Reuters also blocks by default and allows ChatGPT-User and OAI-SearchBot by name. The Guardian lets all three OpenAI bots through and blocks all three of Anthropic's. Spiegel's file even carries a comment saying the OpenAI search crawlers are being let into selected sections as a test.

So "block training, allow retrieval" isn't the whole story for big publishers anymore. It's closer to "block by default, then allow the operators you have a reason to allow". The first version of this audit, from April 20, reported much lower block rates. I can't tell how much of that gap comes from sites tightening their files and how much from the April parsing, so these numbers replace the old ones rather than being compared with them.

Should you block AI crawlers?

Not a yes/no question. It depends on three things: how you make money, whether your content is substitutable, and whether AI citation actually drives traffic to you today.

If you're a news publisher behind a paywall, the logic that drove the NYT lawsuit applies. Training bots learn your content and a few months later the model paraphrases your reporting without sending you a reader. Blocking training is defensive. Allowing retrieval is the open question: you show up as a cited source in ChatGPT and Perplexity answers, but the referral traffic is still a tiny fraction of organic Google. Anthropic's own crawl-to-refer ratio (per Cloudflare's August 2025 data) was roughly 2,500:1 on news sites: they crawl a lot and send little back.

If you run a marketing or lead-gen site, the opposite math applies. You want to be mentioned in ChatGPT answers about your category. Blocking AI bots cuts you out of that channel. Allow everything, including training. The downside of training is minimal when your content is a sales pitch, and the upside of citation is real. Lumina itself takes this stance.

If you're an affiliate site, documentation site, or comparison review, the retrieval angle dominates. People ask ChatGPT "what's the best X" and the answer cites 3-5 sources. If you're not one of them, you don't exist. Allow OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, Perplexity-User, and PerplexityBot. You can block GPTBot and ClaudeBot if you want to push back on training, but the retrieval bots are the ones doing you favors.

Default stance if you're unsure: block GPTBot, ClaudeBot, CCBot and Google-Extended, and allow everything else. Most of the big publishers in the audit above go further and block retrieval bots too.

The robots.txt reference for AI crawlers

Three copy-pasteable configurations, from most permissive to most restrictive. Put the file at /robots.txt at your domain root. User-agent matching is case-insensitive. Rules apply to paths, which are case-sensitive. How RFC 9309 groups and matches rules is in the robots.txt guide.

1. Allow everything (Wikipedia-style)

User-agent: *
Disallow:

Use this if your content is open-licensed, or you actively want citation and don't mind training. Most B2B SaaS and marketing sites should be here.

2. Block training, allow retrieval (the middle ground)

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: AppleBot-Extended
Disallow: /

User-agent: *
Disallow:

This blocks every training bot I've documented and lets retrieval bots through via the * fallback. OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, and Perplexity-User all still have access.

3. Block everything AI (the NYT approach)

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: AppleBot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: *
Allow: /

The explicit Googlebot: Allow matters here. Without it, the list can read as hostile to all bots. You want classic Google search to continue crawling; the block is AI-only. Keep in mind that ChatGPT-User, Perplexity-User and Google-Agent generally don't follow these lines, so this config reduces AI access rather than ending it.

Common mistakes

• Blocking GPTBot but leaving CCBot open. Common Crawl feeds Llama, Falcon, older GPT, and dozens of academic models. If your goal is "no training use of my content," you need to block both.
• Expecting robots.txt to block user-triggered bots. ChatGPT-User, Perplexity-User and Google-Agent generally don't follow it, because a person asked for the page. If you need to keep them out, that takes server-side rules.
• Wildcard blocks catching legitimate bots. A line like User-agent: *bot Disallow: / isn't valid robots.txt syntax and will be ignored. Worse, Disallow: /*bot* can block arbitrary URLs. Use explicit user-agent names.
• Forgetting to deploy. robots.txt lives at the domain root. Subfolder CMSes sometimes generate a robots.txt that never gets served. Test with curl https://yourdomain.com/robots.txt. If that returns 404 or the wrong file, nothing you wrote matters.
• Bumping robots.txt without checking the response. Some CDNs cache robots.txt for 24 hours. If you update rules at 10am and the cache doesn't clear until midnight, you've shipped nothing for 14 hours.

FAQ

Where to start

Five steps, in order: