Salesman Hal

robots.txt for AI crawlers: training, search and live fetching

AI companies send different bots for different jobs: one to collect training data, one to build a search index, and one to fetch a page because a person just asked about you. robots.txt lets you treat each differently. Here are the current names, which ones follow the rules, and policies that keep you out of training without losing customers.

Updated

Three jobs, three kinds of bot

“AI crawler” covers bots doing very different things, and treating them all alike is the most common robots.txt mistake we see.

  • Training crawlers collect pages that may be used to train future models. Blocking them is a statement about how your content is used. It doesn’t change whether assistants can find you today.
  • Search crawlers build the index that AI search answers are drawn from: ChatGPT search, Claude’s web search, Perplexity. Block one and you drop out of that product’s answers.
  • User-triggered fetchers visit a page at the moment a person asks the assistant something (“check this plumber’s hours”, “what does this company charge?”). They don’t crawl; they read one page and answer.

A newer, fourth kind is the hosted browser agent: a real browser running in the AI company’s cloud, clicking through a site to finish a task for someone, such as filling in a booking form. Anthropic calls its version Claude-Agent and Google calls its own Google-Agent. These are covered below too.

The user agents, by company

As of November 2026, from each company’s own documentation:

Name Company Job Follows robots.txt?
GPTBotOpenAITrainingYes; blocking it opts you out of training
OAI-SearchBotOpenAIChatGPT searchYes; changes take about 24 hours
ChatGPT-UserOpenAIFetches for a person, and GPT Actions“Robots.txt rules may not apply”
ClaudeBotAnthropicTrainingYes
Claude-SearchBotAnthropicSearch indexingYes
Claude-UserAnthropicFetches for a personYes
PerplexityBotPerplexitySearch indexing (not model training)Yes; changes take up to 24 hours
Perplexity-UserPerplexityFetches for a person“Generally ignores robots.txt rules”
GooglebotGoogleSearch, including AI Overviews and AI ModeYes
Google-ExtendedGoogleToken only: Gemini training and groundingYes; doesn’t affect Search
Google-AgentGoogleAgents acting on a user’s requestGoogle’s user-triggered fetchers “generally ignore robots.txt rules”
ApplebotAppleSearch for Siri, Spotlight and Safari; data may also train Apple’s modelsYes
Applebot-ExtendedAppleToken only: training opt-outYes; doesn’t crawl and doesn’t affect search
meta-externalagentMetaTraining, and indexing for productsCan be blocked in robots.txt
Meta-WebIndexerMetaMeta AI search and citationsMeta asks you to allow it
meta-externalfetcherMetaFetches for a person“May bypass robots.txt”
CCBotCommon CrawlOpen web archive anyone can downloadYes

A few details that trip people up:

  • Google-Extended isn’t a bot. No request ever arrives with that name. Googlebot does the crawling, and the token tells Google whether that content may be used to train Gemini models and to ground answers in Gemini Apps and on Vertex AI. It doesn’t affect Google Search, and AI Overviews and AI Mode are part of Search. To leave those, Google offers a separate, site-wide “Search generative AI” setting in Search Console.
  • Applebot-Extended works the same way: it doesn’t crawl, it only governs whether what Applebot collected can train Apple’s models. Pages that block it can still appear in Siri, Spotlight and Safari results.
  • Applebot copies Googlebot if you don’t mention it. Apple says that when robots.txt has no Applebot rules but has Googlebot rules, Applebot follows the Googlebot ones.
  • Hosted browser agents use browser user agents. Anthropic’s Claude-Agent and Google’s Google-Agent send a normal browser user agent with the agent’s name added. Anthropic says to match on the Claude-Agent token, check the source IP, and manage it separately from its robots.
  • There are more. OpenAI also runs OAI-AdsBot, which only checks landing pages submitted as ChatGPT ads. Meta’s facebookexternalhit builds link previews. Other assistants, from Amazon, Mistral, DuckDuckGo and others, have their own names; check each vendor’s page before you rely on one.

Do user-triggered fetchers obey robots.txt?

Mostly not, by the vendors’ own description. Their reasoning is that a person asked for the page, so the fetch is closer to someone opening it in a browser than to a crawl.

  • OpenAI: because ChatGPT-User’s actions are started by a user, “robots.txt rules may not apply”. OpenAI also says ChatGPT-User isn’t used to decide whether content appears in search.
  • Perplexity: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”
  • Google: its user-triggered fetchers, which include Google-Agent, “generally ignore robots.txt rules”.
  • Meta: meta-externalfetcher “may bypass robots.txt because it performs fetches that were requested by the user.”
  • Anthropic is the exception. It says its bots honour robots.txt, and that disallowing Claude-User “prevents our system from retrieving your content in response to a user query”.

Two practical consequences. First, a Disallow for ChatGPT-User or Perplexity-User may not stop them, so don’t rely on it for anything private: robots.txt was never access control, and the standard says so. Second, if you do want to stop these fetchers, it takes a firewall or CDN rule, and that’s a blunt tool with a real cost, covered below.

How robots.txt rules are matched

robots.txt is defined by RFC 9309. Three rules from it explain almost every surprise:

  1. A bot uses only the group that names it. If there’s a group for OAI-SearchBot, OAI-SearchBot follows that group and ignores the User-agent: * group completely. Only bots with no group of their own fall back to *.
  2. The longest matching path wins. Allow: /blog/ beats Disallow: / for pages under /blog/. If an Allow and a Disallow are the same length, Allow wins.
  3. Names are matched without regard to case, and several User-agent lines in a row share one set of rules.

The first rule is the dangerous one. Add a group for one AI bot and you’ve silently dropped every rule in your * group for it, so repeat anything that still matters:

User-agent: *
Disallow: /checkout/

# This group replaces the one above for OAI-SearchBot, so the
# checkout rule has to be repeated here.
User-agent: OAI-SearchBot
Disallow: /checkout/
Disallow: /drafts/

Bots cache robots.txt, typically for up to 24 hours, and the file applies per host: shop.example.com needs its own. Anthropic specifically reminds you to add rules on every subdomain you want covered.

Example policies

1. Allow everything

The simplest policy, and the right one for most service businesses that want to be recommended. An empty robots.txt, or none at all, means the same thing; listing your sitemap is a bonus.

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

2. Opt out of training, stay in search and answers

If you don’t want your content used to train models but do want AI assistants to find you, quote you and send people your way, block the training names and leave the rest alone:

# Training crawlers and training-only tokens: opt out.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /

# Everyone else, including AI search crawlers (OAI-SearchBot,
# Claude-SearchBot, PerplexityBot) and fetchers acting for a person
# (ChatGPT-User, Claude-User, Perplexity-User).
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/sitemap.xml

Know the trade-offs before you ship it:

  • Google-Extended covers more than training. It also controls grounding in Gemini Apps, so blocking it may mean Gemini uses your pages less when answering questions. Your Google Search listing, AI Overviews included, is unaffected.
  • meta-externalagent does two jobs. Meta says it crawls for training and to improve products by indexing content. Blocking it may reduce how much Meta’s products know about you; Meta-WebIndexer, its search crawler, stays allowed.
  • Blocking CCBot is forward-looking. Common Crawl’s archive is public, so pages already collected stay in old copies.
  • Copy any rules your * group needs into the training group if they ever start allowing parts of the site, because of the matching rule above.

A standard “no AI training” signal is also on the way, based on IETF drafts from the aipref working group. Anthropic says it began honouring such a signal on October 30, 2026, while still treating rules that name ClaudeBot as more specific. Google says it intends to support it, via user-agent: __usage_ai-training, in early 2027. Until it settles, naming the training bots, as above, is the method every vendor documents.

3. What not to do

Don’t copy a “block all AI” list from a blog post. Most of them include OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User and Perplexity-User, which removes you from AI search and, where the fetcher obeys, from live answers, while doing nothing about training that the training-only names wouldn’t do. And don’t block Googlebot or Applebot to avoid AI: that takes you out of Google Search and Siri altogether.

How blocking can cost you leads

Picture a homeowner asking ChatGPT, Claude or Perplexity: “Find me a plumber near me who can come today and tell me roughly what a leak repair costs.” The assistant searches its index, then fetches a few business sites live to check hours, service area and prices. If it can book or send an inquiry, it may try that too.

Every step depends on a bot you might have blocked:

  • Block the search crawler and you’re not in the shortlist. OpenAI says sites that block OAI-SearchBot won’t be shown in ChatGPT search answers, though they may still appear as navigational links.
  • Block the live fetcher (in robots.txt for Claude-User, or at the firewall for the others) and the assistant can’t check your details. It describes you vaguely or picks a competitor whose page it could read.
  • Block hosted browser agents with a bot wall or CAPTCHA and the agent can’t finish the booking. Google’s guide to allowing Google-Agent warns about exactly this: allow rules need to cover searches, detail pages, forms, booking and the APIs behind them, not just the home page.

None of this shows up in your analytics as a lost lead. The customer never reached your site, and the assistant didn’t tell you it gave up. That’s why we suggest checking robots.txt, your CDN’s bot settings and your security plugin together, and measuring which agents actually reach you. Our guide to measuring AI agent traffic covers how.

Cloudflare’s AI crawler settings

If your site runs through Cloudflare, robots.txt isn’t the only gate. Cloudflare’s AI Crawler and Agent access policies sort AI traffic into the same three groups, Search, Agent (“acting in real time on a person’s behalf”) and Training, and enforce them at the network edge, so they work even on bots that ignore robots.txt.

  • New sites. Since September 15, 2026, Cloudflare proposes recommended settings based on whether the site earns money from ads. For ad-supported sites: Search allowed, training disallowed, and agents blocked on pages with ads. For sites without ads: all three allowed. Cloudflare’s docs, updated in November 2026, say the three policies default to Allow for new domains. Either way, check what your zone actually has under Security, then Settings.
  • “Block” now reaches search. Since the same date, Block and “Block on pages with ads” also apply to mixed-use crawlers such as Googlebot, Applebot and Bingbot. Choose Block for Training and you can drop out of regular search. The “Disallow AI Training” option exists for that: it publishes a no-training preference in your robots.txt and keeps Google, Apple and Microsoft’s search crawlers allowed.
  • Managed robots.txt is being replaced by a feature Cloudflare calls Bot Preference Sync, and sites that used it are migrated automatically.
  • Pay per crawl lets a site charge AI crawlers per request: a crawler either signals that it will pay, or gets an HTTP 402 Payment Required with the price. It’s in closed beta, and Cloudflare says it’s evolving the idea into “Pay Per Use”, paying publishers when their content creates value in an answer.

For a local business, the Agent setting matters most: it covers the fetchers and browser agents that would read your page or book on a customer’s behalf. We go through each setting in Cloudflare and AI agents.

Telling real bots from fakes

Anyone can send User-Agent: GPTBot. Scrapers routinely pretend to be well-known crawlers, and Common Crawl warns that some falsely claim to be CCBot. If you’re making allow or block decisions in a firewall, or reading your logs, check the claim:

  • Published IP ranges. OpenAI publishes one JSON file per bot (openai.com/gptbot.json, searchbot.json, chatgpt-user.json). Perplexity publishes perplexitybot.json and perplexity-user.json. Anthropic lists its bots’ addresses at claude.com/crawling/bots.json and Claude-Agent’s under claude.com/.well-known/sail. Google has separate files for common crawlers, user-triggered fetchers and user-triggered agents. Common Crawl publishes index.commoncrawl.org/ccbot.json. Fetch these automatically; they change.
  • Reverse DNS. Google’s crawlers resolve to googlebot.com hostnames, Applebot to applebot.apple.com, and CCBot to crawl.commoncrawl.org. Look up the IP’s hostname, then confirm the hostname resolves back to the same IP.
  • Web Bot Auth. A newer method built on HTTP Message Signatures (RFC 9421) and IETF drafts. The bot signs each request and sends a Signature-Agent header pointing to its public keys at /.well-known/http-message-signatures-directory, so you can verify it cryptographically instead of trusting an IP. Google is testing it with Google-Agent, signed as https://agent.bot.goog, but signs only some requests, so it says to keep IP and reverse-DNS checks as a fallback. OpenAI publishes a key directory at chatgpt.com. Cloudflare uses Web Bot Auth to verify bots and agents.

One warning from Anthropic: blocking its bots by IP instead of in robots.txt “may not work correctly”, because it stops them reading your robots.txt in the first place. Use robots.txt for preferences, and the firewall for bots that ignore them.

A checklist

  1. Open https://yoursite.com/robots.txt and list every group that names an AI bot.
  2. Decide on training separately from search and answers. If you opt out of training, name only the training bots and tokens.
  3. Make sure OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Applebot aren’t blocked, unless you mean it.
  4. Make sure ChatGPT-User, Claude-User and Perplexity-User aren’t blocked, in robots.txt or your firewall.
  5. Check your CDN, hosting and security-plugin bot settings too. They can block bots that robots.txt allows.
  6. Repeat important * rules inside any bot-specific group, and add the file to every subdomain.
  7. Allow a day for bots to pick up changes, then check your logs or analytics to see who’s visiting.

How Salesman Hal handles AI crawlers

The free AI-readiness scan reads your robots.txt the way RFC 9309 describes, most specific group first and longest rule winning, and checks ten names: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended and Applebot-Extended. It lists each one your file blocks. It counts the training names too, so if you’ve opted out of training on purpose, read those lines as a reminder of the choice, not a fault. Once Hal is installed, its visit analytics recognise AI agents by user agent, through the WordPress plugin or the Cloudflare Worker, and label each visit as training, AI search, or answering a person’s question, so you can see which assistants actually reach your site. Hal doesn’t edit your robots.txt.

See what the report shows

FAQ

If I block GPTBot, will I disappear from ChatGPT?

No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and live page fetches use ChatGPT-User. Blocking GPTBot only signals that your content shouldn’t be used for training.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended doesn’t affect Google Search, and AI Overviews and AI Mode are part of Search. It controls Gemini training and grounding in Gemini Apps and Vertex AI. To manage AI Overviews and AI Mode, Google offers the Search generative AI control in Search Console.

Can I block ChatGPT-User or Perplexity-User with robots.txt?

You can add the rule, but both vendors say their user-triggered fetchers may not follow it. Anthropic says Claude-User does. To actually stop the others you’d need a firewall or CDN rule, which also stops customers’ assistants from reading your site.

Is robots.txt enough to protect private pages?

No. The standard itself says robots.txt rules are not a form of access authorization. Anything private needs a login or another real access control.

How long do changes take?

Bots usually cache robots.txt for up to a day. OpenAI and Perplexity both say about 24 hours for their search crawlers.

Should I just block all AI bots to be safe?

For a business that wins customers through enquiries, usually not. Blocking search crawlers and live fetchers removes you from the answers people now ask assistants for. If training is your concern, block only the training names.

Sources

Further reading

Let the right bots in, then see what they do.

Hal shows which AI agents visit your site and gives them a way to ask questions, send inquiries and book meetings. Set up Hal in about ten minutes. Free during early access.

Or run the free AI-readiness scan first.