Cloudflare and AI agents: let in the ones you want
If your site sits behind Cloudflare, a handful of settings decide whether ChatGPT, Claude and other assistants can read your pages when a customer asks about you. Here’s what each setting does in late 2026, how to keep assistants in while keeping training crawlers out, and how to check it’s working.
Updated
Why this matters for a business site
When someone asks ChatGPT for “a plumber near me who can come tomorrow”, the assistant often fetches pages live to check the details. OpenAI says its ChatGPT-User agent handles these user-initiated actions, and that sites which opt out of OAI-SearchBot won’t be shown in ChatGPT search answers. Other assistants work the same way, each with its own user agents.
Cloudflare sits in front of a large share of business websites, and its bot settings decide whether those requests get your page, a challenge, or a block. A setting turned on years ago to stop scrapers can quietly turn away the assistants that would send you customers. Nobody tells you; the assistant just answers from somewhere else.
Search, Agent and Training
On July 1, 2026 Cloudflare started grouping AI traffic by what it does rather than by who sends it. The setting is called AI Crawler and Agent access policies (it used to be “Configure AI bot policies”). Find it under Security → Settings, filtered to Bot traffic.
| Behaviour | What Cloudflare means by it | Actions |
|---|---|---|
| Search | Crawlers that index your content so they can answer questions about it later. | Allow, Block on pages with ads, Block |
| Agent | Automated activity acting for a person right now, often with that person waiting: chat fetchers such as ChatGPT-User, and browser-use agents. |
Allow, Block on pages with ads, Block |
| Training | Crawlers collecting content to train or fine-tune models, including crawlers that train and search under one name. | Allow, Disallow AI Training, Block on pages with ads, Block |
Each policy covers verified bots with that behaviour and unverified bots that match it. Under Agent you can also set actions for individual Personal Agents; they inherit the Agent policy by default, and Cloudflare recommends leaving them on Allow.
For a new domain, all three start on Allow, unless you tell Cloudflare during onboarding that the site makes money from ads. Then Training starts on Disallow AI Training and Agent on Block on pages with ads.
What happened to “Block AI bots”
Cloudflare is deprecating its original “Block AI bots” switch. Sites that never touched the new controls were moved over automatically on September 15, 2026:
- “Block AI bots” was off: Search, Training and Agent all Allow.
- It was on (either option): Search Allow, Training Disallow AI Training, Agent Block on pages with ads.
The other change in that release matters more. Cloudflare now judges crawlers by everything they do, so Googlebot, Applebot and Bingbot count as Training crawlers too. Block, or Block on pages with ads, on Training now stops them entirely, search included. Disallow AI Training is the gentler option: it publishes a no-training preference in robots.txt, keeps “accountable” mixed-use crawlers (Google, Apple and Microsoft) allowed for search, and blocks every other training crawler, including the training-only crawlers from Amazon, Anthropic, Meta and OpenAI.
One gap: Cloudflare notes that Bing reads training preferences from a noarchive meta tag for now, with robots.txt support targeted for early 2027, so Disallow AI Training doesn’t yet carry a no-training preference to Bing.
Bot Preference Sync and content signals
Bot Preference Sync (launched August 21, 2026, on every plan) writes your Search, Agent and Training choices into robots.txt, so the file and the edge rules say the same thing. It replaces the older managed robots.txt and is on by default for new customers. A few things to know:
- It adds lines only when at least one policy is restrictive. If everything is Allow, your robots.txt is served unchanged.
- If your origin has a robots.txt, Cloudflare puts its block in front of yours and serves both as one file. If you have none, it creates one.
- It works by category and doesn’t read your custom rules. If you want per-crawler rules in robots.txt, turn sync off and write the file yourself.
- Changes go live within seconds, without a cache purge.
The block it writes includes a Content-Signal line from Cloudflare’s Content Signals Policy. There are three signals: search (building a search index and showing links and short excerpts, not AI summaries), ai-input (feeding content to a model in real time, as in AI answers) and ai-train (training or fine-tuning). If you write your own robots.txt, the same line works there:
User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Like everything in robots.txt, these are requests, not locks. Crawlers can ignore them, which is why Cloudflare also enforces Block and Disallow at the edge. Our robots.txt guide lists the AI user agents worth knowing by name.
Verified bots, signed agents and Web Bot Auth
Anyone can put “ChatGPT-User” in a user agent, so Cloudflare keeps a list of verified bots. To be verified, a bot has to prove who it is, either with a published IP list and stable user agent, reverse DNS, or a cryptographic Web Bot Auth signature, and has to behave: follow robots.txt, keep request rates reasonable and not dodge site owners’ preferences.
Web Bot Auth signs each request with HTTP Message Signatures (RFC 9421). The request carries Signature and Signature-Input headers plus a Signature-Agent header pointing at the operator’s public keys, published at /.well-known/http-message-signatures-directory. Cloudflare checks the signature for you. The scheme rests on IETF drafts, not finished standards.
Cloudflare used to talk about “signed agents” as a separate group. Since July 1, 2026 the difference is a field in its bot directory: Direct bots are run by one operator, while Intermediary ones are services many people drive, such as an agent platform. You might trust the operator but not every person using it, so Cloudflare is experimenting with the Forwarded header to pass on who’s responsible for a request.
Two consequences for site owners. First, “verified” no longer means “allowed”: the behaviour policy decides access. Second, Cloudflare’s Free plan spots AI crawlers by user agent only, so it sees the well-known ones that name themselves; paid plans add Cloudflare’s bot detection for a more thorough picture.
Bot Fight Mode side effects
Bot Fight Mode is the free bot setting. It challenges traffic that looks like known bot patterns with expensive challenges. Cloudflare says verified bots have historically been left out of default bot configurations, but plenty of agent traffic isn’t verified: smaller assistants, new agents and browser-use agents running in the cloud. Those can get a challenge they can’t solve.
The catch is that Bot Fight Mode can’t be tuned. Cloudflare’s docs say you can’t bypass or skip it with WAF custom rules or Page Rules; it runs in its own pipeline where Skip and Allow have no effect. The one exception is an IP Access rule that matches first, which is no help for agents that come from changing addresses.
Super Bot Fight Mode (Pro, Business and Enterprise) is more flexible. It groups traffic into Definitely automated, Likely automated and Verified bots, with an action for each, and it runs on Cloudflare’s rules engine, so a WAF custom rule with a Skip action can exempt traffic you trust. Cloudflare warns that both products may challenge API and mobile-app traffic too.
To see whether this is happening, open Security → Analytics → Events and look for challenged requests whose service is “Bot Fight Mode”, or the Super Bot Fight Mode equivalent, with an assistant’s user agent. A quick test from your own computer shows whether a user-agent rule is in the way (it won’t pass as verified, so it can’t prove an agent gets through):
# Ask for your home page the way two assistants' fetchers identify themselves.
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0; compatible; ChatGPT-User/1.0" https://example.com/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0; compatible; Claude-User/1.0" https://example.com/
The setup we recommend
For a business that wants assistants to find it, answer questions about it and send it customers:
- Search: Allow. Blocking it takes you out of AI search and, now, ordinary search too.
- Agent: Allow. This is the person asking ChatGPT or Claude about you right now. Check that the Personal Agents under it are on Allow too.
- Training: your call. If you’d rather not feed model training, choose Disallow AI Training, not Block. Block on Training removes Googlebot, Applebot and Bingbot from your site.
- If your site shows ads, remember that “Block on pages with ads” on Agent turns assistants away from every page with an ad on it.
- Leave Bot Preference Sync on unless you maintain robots.txt by hand, and read the result at
/robots.txtto be sure. - Check Bot Fight Mode. If events show it challenging assistants, weigh turning it off on Free, or move to Super Bot Fight Mode, set Verified bots to Allow and add Skip rules where you need them.
- Check custom rules. Old WAF rules that block by user agent (“block anything with bot in it”) catch assistants too.
Reading AI Crawl Control
AI Crawl Control, renamed from AI Audit, is on every plan with nothing to set up. It has three main views:
- Overview: a summary, your managed robots.txt status, and trends for total, allowed and unsuccessful requests, grouped by operator.
- Crawlers: each crawler’s operator, category (AI crawler, AI assistant, archiver and so on), requests, data transfer and an Allow or Block action. Allowing a crawler here doesn’t exempt it from robots.txt.
- Metrics: requests over time grouped by crawler, category, operator or host; the status codes crawlers got back; the content types they asked for and received; the most popular paths; and, on paid plans, referrals, meaning visits AI services sent you.
Questions worth asking of it each month:
- Are assistants getting 200s? Filter to AI assistant crawlers and look at the status code chart. A run of 403s means something is blocking them.
- What are they reading? Popular paths show whether assistants find your service, pricing and contact pages or only the home page.
- Who ignores robots.txt? The robots.txt (Directives) tab lists violations: the crawler, the path and the line it ignored. It also shows whether crawlers can fetch your robots.txt at all, and whether it carries content signals. Violations are worked out against your current file, so a new rule can flag older visits that were fine when they happened.
Blocks you set in AI Crawl Control are a WAF custom rule Cloudflare manages for you; you can extend it in the WAF for path-based exceptions. On paid plans you can choose whether blocked crawlers get 403 Forbidden or 402 Payment Required, with your own message.
What AI Crawl Control can’t tell you is what happened next: whether an assistant recommended you, or a visit turned into an inquiry. For that you need your own logs and lead tracking; see our guide to measuring AI agent traffic.
Pay per crawl and Markdown for Agents
Pay per crawl lets a site charge AI crawlers per page, with Cloudflare as merchant of record. A crawler either signals that it will pay and gets the page, or gets 402 Payment Required with the price. It’s in closed beta (sign up, or ask your account executive on Enterprise), and any WAF or bot rule that blocks a crawler overrides it. It’s aimed at publishers. A business that wants customers from assistants usually wants them reading for free.
Markdown for Agents converts your HTML to Markdown at the edge when a client sends Accept: text/markdown, stripping navigation and scripts and keeping your JSON-LD. It’s free on Pro, Business and Enterprise (not Free), under Agent Readiness → Markdown for Agents. Note that if your origin doesn’t send a content-signal header, Cloudflare adds one to these responses saying ai-train=yes, search=yes, ai-input=yes, so set your own if you’ve disallowed training.
Workers: discovery files and visit logs
A Cloudflare Worker is a small script that runs on Cloudflare’s network in front of your site. It’s a neat way to add things for AI agents without touching your origin: serve the discovery files that point assistants at your MCP server (see our MCP discovery guide), serve an llms.txt, add a tag to every page’s head, and record which assistants visited which pages. A minimal version:
const AGENTS = ['ChatGPT-User', 'OAI-SearchBot', 'Claude-User', 'PerplexityBot'];
const MCP_URL = 'https://mcp.example.com/mcp';
export default {
async fetch(request, env, ctx) {
// If this code throws, send the request to the origin as if we weren't here.
ctx.passThroughOnException();
const url = new URL(request.url);
const userAgent = request.headers.get('user-agent') ?? '';
const agent = AGENTS.find((token) => userAgent.includes(token));
if (agent) {
// Runs after the response is sent, so logging never slows the visit.
ctx.waitUntil(fetch(env.LOG_URL, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ agent, path: url.pathname, at: new Date().toISOString() }),
}).catch(() => {}));
}
if (url.pathname === '/.well-known/mcp.json') {
return Response.json({ servers: [{ url: MCP_URL, transport: 'streamable-http' }] });
}
return fetch(request);
},
};
Things that make the difference between helpful and harmful:
- Fail open.
ctx.passThroughOnException()sends the request to your origin if your code throws. It doesn’t cover a Worker that runs out of CPU or memory, so keep the work small. - Log after the response. Work passed to
ctx.waitUntil()runs after the visitor has their page, with a 30-second limit. Never make the visitor wait for your logging. - Mind the free quota. The Workers Free plan allows 100,000 requests a day, reset at midnight UTC. After that, a route set to fail open skips the Worker; one set to fail closed shows visitors Cloudflare error 1027. A Worker on every page of a busy site should fail open, or run on the $5-a-month paid plan.
- User agents are a claim. Matching on the user agent tells you what a request says it is. That’s fine for counting visits, but don’t use it to grant access.
- Only what reaches the Worker gets logged. A request Cloudflare blocked or challenged doesn’t produce your page, so check AI Crawl Control and Security Events for those.
What differs by plan
| Feature | Free | Paid plans |
|---|---|---|
| Search, Agent and Training policies; Bot Preference Sync | Yes | Yes |
| AI Crawl Control | Yes; crawlers identified by user agent | Adds bot-detection IDs, a custom block response (403 or 402) and referral data |
| Bot protection | Bot Fight Mode; no exceptions | Super Bot Fight Mode (Pro, Business, Enterprise); Skip rules work |
| Markdown for Agents | No | Pro, Business, Enterprise |
| Pay per crawl | Closed beta | |
| Workers | 100,000 requests a day | From $5 a month for the account, 10 million requests included |
How Salesman Hal handles Cloudflare
If your site is on Cloudflare, you can install Hal by clicking Connect Cloudflare in the dashboard and approving access to read your zones and edit Workers scripts and routes. No passwords or API tokens change hands, which means an agency can set it up without the client’s Cloudflare login. Hal uploads its Worker to your account and attaches it to every request on the domain. The Worker serves your MCP discovery files (/.well-known/mcp, /.well-known/mcp.json and the server card), adds a <link rel="mcp"> tag and Hal’s WebMCP script to your HTML pages, and logs visits from known AI agents to your Hal dashboard after each response is sent. Everything else goes to your origin untouched, and if Hal’s code fails, visitors get your page as if Hal weren’t there. Hal won’t take over a route another Worker already handles, and you can remove it from the Hal dashboard. Hal doesn’t change your bot settings, so the recommendations above are still yours to make.
FAQ
I turned on “Block AI bots” years ago. Am I blocking ChatGPT?
If you never touched the newer controls, Cloudflare moved you to Search Allow, Training Disallow AI Training and Agent Block on pages with ads. Assistants can reach pages without ads, but check Security → Settings → AI Crawler and Agent access policies to be sure, and set Agent to Allow if you want them everywhere.
Will blocking training hurt my Google ranking?
Disallow AI Training shouldn’t: Cloudflare keeps Googlebot, Applebot and Bingbot allowed for search under it. Block on Training is different; it now stops those crawlers entirely, search included.
Can I allow ChatGPT but block GPTBot?
Yes. ChatGPT-User (live fetches for a person) falls under Agent, OAI-SearchBot under Search and GPTBot under Training, so Allow, Allow and Disallow AI Training does exactly that. In AI Crawl Control you can also allow or block individual crawlers.
Does robots.txt stop AI crawlers?
Only the ones that choose to obey it. Cloudflare’s Block and Disallow settings enforce your choice at the edge for the crawlers it recognises, and the robots.txt tab in AI Crawl Control shows who ignores the file.
Should I turn off Bot Fight Mode?
Only if it’s challenging traffic you want. Check Security Events for Bot Fight Mode challenges on assistant user agents first. On Free there’s no way to make exceptions, so the choice is on or off; on Pro and up, Super Bot Fight Mode lets you skip trusted traffic instead.
Do I need a Worker to see AI visits?
No. AI Crawl Control shows AI crawler traffic on every plan. A Worker adds what Cloudflare doesn’t: your own record of each visit, joined up with the inquiries and bookings that follow, plus discovery files and head tags without changing your site.
Sources
- AI Crawl Control, Cloudflare Docs
- Manage AI crawlers, Cloudflare Docs
- Analyze AI traffic, Cloudflare Docs
- Track robots.txt, Cloudflare Docs
- What is pay per crawl?, Cloudflare Docs
- AI Crawler and Agent access policies, Cloudflare Docs
- Managed robots.txt (Bot Preference Sync), Cloudflare Docs
- Signed agents and verified bots, Cloudflare Docs
- Web Bot Auth, Cloudflare Docs
- Bot Fight Mode, Cloudflare Docs
- Super Bot Fight Mode, Cloudflare Docs
- Markdown for Agents, Cloudflare Docs
- Context (ctx): waitUntil and passThroughOnException, Cloudflare Workers Docs
- Workers limits, Cloudflare Workers Docs
- Workers pricing, Cloudflare Workers Docs
- Your site, your rules: new AI traffic options for all customers, Cloudflare blog, July 1, 2026
- Say it once: introducing Bot Preference Sync, Cloudflare blog, August 21, 2026
- Have it both ways: stay discoverable in search while disallowing AI training, Cloudflare blog, September 15, 2026
- Overview of OpenAI crawlers, OpenAI
Further reading
- Phases list (the order Cloudflare’s security rules run in), Cloudflare Docs
- robots.txt for AI crawlers, Salesman Hal
- WebMCP: a practical guide for business websites, Salesman Hal
On Cloudflare? Hal installs in one click.
Connect Cloudflare and Hal adds an MCP server, discovery files, WebMCP tools and AI visit logs to your site. Free during early access.
Or run the free AI-readiness scan to see what assistants can reach today.