AI Crawlers: What They Do and How to Control Them in 2026

An AI crawler is a bot that fetches web pages so a large language model can do something with them. In 2026 that something is one of three jobs: train the next model, populate a search index, or answer a specific user's question live. Each job has its own bot. The gap between "allow everything" and "block everything" is the part most robots.txt guides skip, and it's where the interesting decisions live.

This guide is the full reference: who the crawlers are, what they actually do, how much traffic they generate, and what 10 top news sites have decided — with a live audit done the morning of publication. The canonical companion tool is Lumina's Crawler Access Checker, which runs the same analysis against any URL.

What is an AI crawler?

An AI crawler fetches HTML and sends it downstream to a model that will either memorize it, index it, or quote it. The bot itself looks identical to a classic web crawler — same HTTP requests, same user-agent headers, same respect (usually) for robots.txt. What changes is what happens to the content after it's fetched.

Classic crawlers like Googlebot exist to rank pages against each other and return links in a search result. Users click through. AI crawlers skip the click. ChatGPT reads your page, reasons about the content, and writes an answer that may cite you or may not. That's the shift. Your content stops being a destination and becomes a source.

Because "AI crawler" is a job description rather than a strict category, the list evolves fast. The bots from OpenAI, Anthropic, Google, Perplexity, Apple, Meta, and Common Crawl are the ones that matter in 2026. Vercel's own edge logs (late 2024) show GPTBot, Claude, AppleBot, and PerplexityBot accounting for the overwhelming majority of declared AI-crawler volume across their network. A few dozen smaller crawlers exist, but these top bots are where the interesting decisions live.

The major AI crawlers in 2026

Every public-facing AI system worth tracking declares a user agent. Here are the ones you'll see in your server logs:

Two things worth noting. First, Googlebot itself isn't on this list — Googlebot still fetches your pages for classic search, and Google-Extended is the separate opt-out you set to exclude your content from Gemini training. Blocking Google-Extended doesn't affect your ranking in Google Search. Second, Bingbot isn't on this list either, even though ChatGPT Search partially relies on Bing's index. Microsoft has signaled that blocking Bingbot hurts both traditional search and AI search — the two share infrastructure there.

Training vs. search: the split that matters

The single biggest 2024 to 2026 shift: OpenAI, Anthropic, and Perplexity all split their one bot into multiple bots with distinct jobs. This is the decision point most robots.txt guides still write around, and it's the one that changes what blocking actually costs you.

OpenAI separated GPTBot from OAI-SearchBot and ChatGPT-User in 2024. Anthropic mirrored the move with Claude-SearchBot and Claude-User. Perplexity runs a two-tier version with PerplexityBot for indexing and Perplexity-User for live retrieval. The pattern is the same everywhere: one bot for training, one for indexing, one for live user-triggered fetches.

Here's why that matters. If you block GPTBot only, you've stopped OpenAI from training on your content — but ChatGPT can still cite you when a user asks it a question, because ChatGPT-User is a different bot. Most publishers want that middle ground: no training, yes citation.

One wrinkle worth knowing. In a December 2025 documentation update, OpenAI removed the robots.txt compliance language from ChatGPT-User and added a note that because those fetches are user-initiated, "robots.txt rules may not apply". Blocking ChatGPT-User in robots.txt no longer works reliably. For training (GPTBot) and search indexing (OAI-SearchBot), robots.txt still does what you'd expect. Anthropic's three bots all still respect robots.txt per their support docs. So the training-vs-retrieval split is still a valid strategy, with a small asterisk on ChatGPT-User specifically.

Concrete example. The Guardian allows OAI-SearchBot and ChatGPT-User through the * fallback while blocking ClaudeBot and CCBot outright. They want to be cited in ChatGPT's live answers but not used in training corpora. The NYT takes the opposite approach: after their 2023 lawsuit against OpenAI, they block every AI bot in existence. Wikipedia takes the third stance — allow everything, because their CC BY-SA license already permits re-use.

How much traffic AI crawlers generate

Cloudflare reported roughly 50 billion AI crawler requests per day across their network in 2025 — about 1% of all web traffic routed through them, and sharply up from 2024. Most of that is training, not answering user queries: Cloudflare's own breakdown shows training-purpose crawls making up nearly 80% of AI bot volume, with GPTBot and ClaudeBot together accounting for about half of all observed AI crawling.

Classic search still dwarfs the AI side. Vercel's edge logs for late 2024 put Googlebot at 4.5 billion requests per month versus GPTBot at 569 million, Claude at 370 million, AppleBot at 314 million, and PerplexityBot at 24 million. That's roughly an 8× gap to GPTBot and nearly 200× to PerplexityBot. AI crawlers are growing fast but haven't caught traditional search in raw volume, and probably won't for years.

Not every crawler follows the rules, either. Per Vercel's own data the major declared bots — GPTBot, Claude, AppleBot — respect robots.txt. The problem is the undeclared ones. A Wired investigation in June 2024 caught Perplexity bypassing robots.txt via an unnamed AWS-hosted crawler; Cloudflare publicly called out the same pattern in August 2025. If a crawler has decided to scrape you, robots.txt won't stop it. For real enforcement you need the WAF or IP-block layer.

Live audit: what 10 top sites actually block

I pulled robots.txt from 10 high-traffic sites on the morning of publication and parsed each for Allow/Disallow directives across 12 AI bots. The pattern is clear: training bots get blocked roughly three times as often as retrieval bots. Here's what the numbers say.

The second-order finding: of the 10 sites, 8 use the three-tier approach (block training, allow retrieval, allow user browsing). Only NYT is all-blocks and only Wikipedia is all-allows. The middle-ground strategy is now the mainstream-publisher default.

Should you block AI crawlers?

Not a yes/no question. It depends on three things: how you make money, whether your content is substitutable, and whether AI citation actually drives traffic to you today.

If you're a news publisher behind a paywall, the logic that drove the NYT lawsuit applies. Training bots learn your content and a few months later the model paraphrases your reporting without sending you a reader. Blocking training is defensive. Allowing retrieval is the open question: you show up as a cited source in ChatGPT and Perplexity answers, but the referral traffic is still a tiny fraction of organic Google. Anthropic's own crawl-to-refer ratio (per Cloudflare's August 2025 data) was roughly 2,500:1 on news sites — they crawl a lot and send little back.

If you run a marketing or lead-gen site, the opposite math applies. You want to be mentioned in ChatGPT answers about your category. Blocking AI bots cuts you out of that channel. Allow everything, including training — the downside of training is minimal when your content is a sales pitch, and the upside of citation is real. Lumina itself takes this stance.

If you're an affiliate site, documentation site, or comparison review, the retrieval angle dominates. People ask ChatGPT "what's the best X" and the answer cites 3-5 sources. If you're not one of them, you don't exist. Allow OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, Perplexity-User, and Perplexity-Bot. You can block GPTBot and ClaudeBot if you want to push back on training, but the retrieval bots are the ones doing you favors.

Default stance if you're unsure: block GPTBot, ClaudeBot, CCBot, and Google-Extended. Allow everything else. This is the middle position and it mirrors what most mainstream publishers have landed on.

The robots.txt reference for AI crawlers

Three copy-pasteable configurations, from most permissive to most restrictive. Put the file at /robots.txt at your domain root. User-agent matching is case-insensitive. Rules apply to paths, which are case-sensitive.

1. Allow everything (Wikipedia-style)

User-agent: *
Disallow:

Use this if your content is open-licensed, or you actively want citation and don't mind training. Most B2B SaaS and marketing sites should be here.

2. Block training, allow retrieval (the middle ground)

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: AppleBot-Extended
Disallow: /

User-agent: *
Disallow:

This blocks every training bot we've documented and lets retrieval bots through via the * fallback. OAI-SearchBot, Claude-SearchBot, ChatGPT-User, Claude-User, PerplexityBot, and Perplexity-User all still have access.

3. Block everything AI (the NYT approach)

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: AppleBot-Extended
Disallow: /
User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Googlebot
Allow: /

User-agent: *
Allow: /

The explicit Googlebot: Allow matters here. Without it, the list can read as hostile to all bots. You want classic Google search to continue crawling — the block is AI-only.

Common mistakes

Six patterns I see in client audits and competitor robots.txt inspections:

• Blocking GPTBot but leaving CCBot open. Common Crawl feeds Llama, Falcon, older GPT, and dozens of academic models. If your goal is "no training use of my content," you need to block both.
• Blocking ChatGPT-User. This removes you from ChatGPT's live answer citations. If that's the goal, fine. If you only wanted to block training, you blocked the wrong bot.
• Wildcard blocks catching legitimate bots. A line like User-agent: *bot Disallow: / isn't valid robots.txt syntax and will be ignored. Worse, Disallow: /*bot* can block arbitrary URLs. Use explicit user-agent names.
• Trusting robots.txt as enforcement. The crawlers you most want to stop are the ones most likely to ignore robots.txt — Perplexity's unnamed AWS crawler, scrapers with faked user agents, undeclared bots. The polite bots respect it. The aggressive ones don't. Layer in WAF or IP blocks if enforcement matters.
• Forgetting to deploy. robots.txt lives at the domain root. Subfolder CMSes sometimes generate a robots.txt that never gets served. Test with curl https://yourdomain.com/robots.txt — if that returns 404 or the wrong file, nothing you wrote matters.
• Bumping robots.txt without checking the response. Some CDNs cache robots.txt for 24 hours. If you update rules at 10am and the cache doesn't clear until midnight, you've shipped nothing for 14 hours.

FAQ

Where to start

If you want working AI-crawler rules on your site this week, do these five things in order: