How to measure AI agent traffic on your website
AI assistants read your site every day, but most of those visits never show up in Google Analytics. Here’s what kinds of AI visitors there are, where each one leaves a trace, how to tell real ones from fakes, and how to turn it all into a monthly report someone will actually read.
Updated
The four kinds of AI visitor
“AI traffic” lumps together visitors that behave very differently and mean very different things for a business. Split them by why they came.
| Kind | Examples | What it means for you |
|---|---|---|
| Training crawlers | GPTBot, ClaudeBot, CCBot, meta-externalagent |
Collecting text that may train future models. No person is waiting on the answer. |
| Search indexers | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
Building the index an assistant searches when it answers. Being in it is how you get cited. |
| User-triggered fetchers | ChatGPT-User, Claude-User, Perplexity-User, Google-Agent |
An assistant reading your page right now because someone asked it something. The closest thing to a prospect. |
| Browser agents | ChatGPT agent’s cloud browser; agent modes in AI browsers such as ChatGPT Atlas and Perplexity Comet; assistants built into Chrome | Loading pages, clicking and filling in forms on someone’s behalf. Often indistinguishable from a person. |
The operators document their own bots. OpenAI says ChatGPT-User handles “user-initiated actions” and isn’t used for automatic crawling; Anthropic describes Claude-User as fetching pages when people ask Claude questions; Perplexity says Perplexity-User visits a page to help answer a question and may link to it. User-triggered fetchers generally don’t follow robots.txt, because a person asked for the page. Google’s Google-Extended is a robots.txt token, not a crawler: Google says it has no user-agent string of its own, so you’ll never see it in a log.
Volumes are lopsided. Cloudflare reported in August 2025 that, across its network over the previous twelve months, about 80% of AI crawling was for training, 18% for search and 2% for user actions. Small numbers of user-triggered fetches can still matter most, because each one is a real question about a business like yours.
Why GA4 misses most of them
Google Analytics counts a visit when its JavaScript tag runs in a browser. That rules out most AI visitors for three reasons:
- Most AI visitors don’t run JavaScript analytics. A crawler or fetcher that downloads your HTML to read the text never executes the tag, so there’s nothing for GA4 to count.
- GA4 drops the ones it recognizes. Google says traffic from known bots and spiders is excluded automatically, using its own research and the IAB’s International Spiders and Bots List, and that “you cannot disable known bot traffic exclusion or see how much known bot traffic was excluded.”
- Browser agents look human. An agent working in a real browser runs your scripts like any visitor. Simon Willison found that ChatGPT Atlas sent the same user-agent string as Chrome on macOS. Those sessions land in GA4 as ordinary Chrome traffic, with nothing marking them as an agent.
So GA4 is the right tool for one slice, people who click through from an AI answer, and the wrong tool for counting AI visits themselves. For those you need logs.
Server and CDN logs
Every request to your server, including from bots that never run JavaScript, can be written to an access log with its time, path, status code, IP address and user-agent string. Depending on how your site is hosted, logs live in one or more places:
- Your web server or host. Apache and nginx write access logs by default; managed WordPress hosts usually let you download them from their control panel, sometimes for only a few days.
- Your CDN. If a CDN serves cached pages, it sees requests your server never does. Cloudflare’s AI Crawl Control dashboard, for example, breaks AI crawler requests down by crawler, operator, status code and path, and shows referrals from AI platforms on paid plans.
- Your application. A plugin or middleware can log only the requests whose user agent matches a known AI agent, which keeps the data small and leaves ordinary visitors out of it.
With a raw access log, two one-liners get you most of the way. The first counts requests per agent for a month:
# Requests per AI agent in one month of an access log (combined log format).
grep '/Nov/2026:' access.log \
| sed -nE 's/.*(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Agent|CCBot|meta-externalagent|meta-externalfetcher|Amazonbot|Bytespider).*/\1/p' \
| sort | uniq -c | sort -rn
The second lists the pages an assistant fetched for its users, which tells you what people were asking about:
# The pages ChatGPT fetched to answer people's questions, most fetched first.
grep 'ChatGPT-User' access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
Keep the list of tokens in one place and review it every few months; new agents appear and version numbers in the strings change.
User-agent matching and its limits
Matching on user agent is how almost every AI traffic report works, and it’s good enough for trends. Know where it breaks:
- Match the token, not the whole string. OpenAI’s strings look like
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, and the version number changes. MatchGPTBoton a word boundary. - Anyone can send any user agent. Cloudflare puts it bluntly: user-agent headers “are easily spoofed.” Scrapers often pretend to be well-known bots to slip past firewalls, which inflates your counts.
- Some traffic doesn’t say what it is. In August 2025 Cloudflare reported that requests it attributed to Perplexity used a generic Chrome-on-macOS user agent and IP addresses outside Perplexity’s published ranges after its declared bots were blocked. Perplexity disputed the report. Either way, undeclared traffic is invisible to user-agent matching.
- Browser agents usually don’t identify themselves in the user agent at all, as above.
So treat user-agent counts as “requests that said they were AI agents.” For anything you’ll act on, such as allowing a bot through a firewall, verify it.
Verifying that a bot is real
There are two ways to check that a request really came from the company it names.
Published IP ranges and reverse DNS
The major operators publish the addresses their bots use as JSON files: OpenAI has one per bot (openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json), Anthropic publishes claude.com/crawling/bots.json, and Perplexity publishes perplexity.ai/perplexitybot.json and perplexity.ai/perplexity-user.json. Check the IP in your log against the matching list:
# Check that a request claiming to be ChatGPT-User came from OpenAI's published ranges.
import ipaddress, json, urllib.request
ranges = json.load(urllib.request.urlopen('https://openai.com/chatgpt-user.json'))
networks = [ipaddress.ip_network(p['ipv4Prefix']) for p in ranges['prefixes'] if 'ipv4Prefix' in p]
def from_openai(ip):
address = ipaddress.ip_address(ip)
return any(address in network for network in networks)
print(from_openai('203.0.113.7')) # False: a documentation-only address
Google also publishes IP lists (including one for user-triggered agents) and supports a reverse DNS check: look up the hostname for the IP, confirm it ends in googlebot.com, google.com or googleusercontent.com, then look the hostname up again and confirm it returns the same IP.
host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
IP checks have limits. Cloudflare notes that cloud addresses are shared and change over time, so the logic tying an IP range to a product “is brittle.” Download the lists fresh rather than hard-coding them.
Signed requests: Web Bot Auth
The newer approach is for the agent to sign each request cryptographically. Web Bot Auth builds on HTTP Message Signatures (RFC 9421): the request carries Signature and Signature-Input headers, plus a Signature-Agent header pointing to the operator’s public keys at /.well-known/http-message-signatures-directory. A simplified example:
Signature-Agent: "https://chatgpt.com"
Signature-Input: sig1=("@authority" "signature-agent");created=1795000000;expires=1795000060;keyid="…";tag="web-bot-auth"
Signature: sig1=:…:
OpenAI signs requests from ChatGPT’s cloud browser this way, with Signature-Agent set to "https://chatgpt.com"; it says this applies to its cloud browser, not its crawlers. Google says it is experimenting with Web Bot Auth for Google-Agent, using the identity https://agent.bot.goog. The bot-specific parts are still IETF drafts, so expect changes.
You rarely need to verify signatures yourself. OpenAI lists Akamai, Cloudflare, HUMAN and Vercel as verifying them automatically; on Cloudflare, a verified request sets the cf.bot_management.verified_bot field, and ChatGPT agent appears as a verified bot. If your site sits behind a proxy, make sure it passes the three signature headers through untouched.
AI referral traffic
Referrals are people, not bots: someone reads an AI answer and clicks a link to your site. They do run your analytics tag, so this is where GA4 earns its place.
- GA4’s AI Assistant channel. GA4’s default channel group now includes “AI Assistant,” for users who arrive from sources such as ChatGPT, Gemini, DeepSeek, Copilot or Grok. Google says AI Overviews and AI Mode are not part of it; that traffic counts as Google search.
- UTM tags. OpenAI’s publisher FAQ says ChatGPT adds
utm_source=chatgpt.comto referral links, so those visits showchatgpt.comas the source even when the referrer is missing. - Referrer stripping. Not every click carries a referrer. Cloudflare noted that traffic referred by Claude’s native app doesn’t include a
Refererheader, and believes the same is true of other native apps. Those visits look like direct traffic, so AI referral numbers are a floor, not a total.
Recipe: AI referrals in a GA4 exploration. In GA4, go to Explore, start a Free form exploration, add the dimensions Session source and Landing page + query string, and the metrics Sessions and Key events. Add a filter on Session source using a regular expression:
.*(chatgpt\.com|chat\.openai\.com|perplexity\.ai|claude\.ai|gemini\.google\.com|copilot\.microsoft\.com|deepseek\.com|grok\.com).*
The leading and trailing .* make the pattern work whether the filter does a full or partial match. Compare the result with the AI Assistant channel; if your regex finds sources the channel doesn’t, consider a custom channel group so the numbers agree month to month. Set up key events (form submissions, bookings, calls) first, or you’ll count visits with no idea whether they turned into anything.
Why caches hide visits from WordPress
On a fast WordPress site, most page views never reach WordPress. A page-cache plugin or the host’s cache serves a saved copy of the HTML, and a CDN may answer from its edge before the request gets to your server at all. That’s great for speed and bad for any plugin that counts AI agents from inside WordPress, because PHP never runs for those requests.
There are three ways around it:
- Log where every request passes. Cloudflare documents that Workers “run before the cache,” so a Worker on your domain sees requests whether or not the page comes from cache. CDN analytics, like AI Crawl Control, see them too.
- Exclude AI agents from the cache. Some cache plugins can bypass the cache for given user agents. It works, but costs server load, and you have to keep the list current.
- Add a pixel that is never cached, as below.
Tracking pixels
A tracking pixel is a 1×1 image in the page. Agents that render pages, such as browser agents, request it along with the rest of the page, and because the image is served with no-cache headers (ideally from a different host), the request is logged even when the page itself came from cache. It doesn’t need JavaScript, so it also catches renderers that load images but skip analytics scripts.
Three rules make a pixel useful:
- Send
Cache-Control: no-store, so no cache answers for it. - Pass the page path in the image URL (
?p=/pricing), so you know which page was read; theRefererheader isn’t always there. - Log only requests whose user agent matches a known AI agent, so you aren’t collecting data about people.
A pixel can’t see everything: a fetcher that only reads the HTML never requests the image. Use it alongside logs, not instead of them.
Turning it into a monthly report
A pile of log lines isn’t a report. A useful monthly report fits on one page, uses the same definitions every month, and compares with the month before. A structure that works:
- Headline numbers: AI agent visits, split into assistant fetches, search indexing and training crawls, each with the change from last month.
- Who came: the top agents by visits, with their purpose.
- What they read: the top pages for user-triggered fetchers. These are the topics people are asking assistants about.
- Referrals: AI Assistant sessions from GA4 and the key events they led to, with a note that apps without referrers aren’t included.
- Outcomes: inquiries, quote requests and bookings that came from AI, if you can identify them.
- Method notes: anything that changed, such as new agents added to the list, a cache setting, or a CDN rule that blocked a bot. Changes in method explain more jumps than changes in the world.
Agencies reporting this to clients will want more on the outcome side; our guide to AEO reporting beyond mentions covers that. And if a bot is missing entirely, check your rules first: the robots.txt guide and the Cloudflare guide cover the settings that most often block AI agents by accident.
How Salesman Hal handles AI agent traffic
Hal logs AI agent visits on the server, so it doesn’t depend on JavaScript, referrers or cookies. The WordPress plugin checks each request against a list of known AI agents and logs only the matches; the Cloudflare Worker does the same at the edge, ahead of the cache; and a no-cache pixel served from Hal’s own domain catches agents that load images on cached pages. Every visit is classed by purpose (answering someone’s question, indexing for AI search, or collecting training data), and the dashboard shows visits by agent, the pages they read, and what agents did through Hal’s MCP server: questions asked, searches, inquiries, quote requests and bookings. A monthly email report compares each number with the month before and lists the questions Hal couldn’t answer.
Hal identifies agents by user agent and doesn’t yet verify signatures or IP ranges, and it doesn’t track referral clicks, so keep GA4 for those. See what Hal reports for agencies, or scan a site free to see whether AI agents can reach it.
FAQ
Can I see AI crawler visits in Google Analytics?
Mostly no. Crawlers and fetchers don’t run the GA4 tag, and Google excludes known bots automatically without telling you how many. GA4 is useful for AI referrals, the people who click a link in an AI answer, through its AI Assistant channel.
Which AI visits matter most to a business?
User-triggered fetches such as ChatGPT-User, Claude-User and Perplexity-User, because each one means a person asked an assistant something and it came to your site for the answer. Search indexers matter for being cited later. Training crawls matter least for leads.
How do I know a “GPTBot” request is really OpenAI?
Check its IP address against the list OpenAI publishes for that bot, or, if your CDN verifies bots, check its verified-bot flag. A user-agent string alone proves nothing.
Why are my AI visit numbers lower than my CDN’s?
Usually because a page cache or CDN served those requests and your server never saw them. Count at the edge, add an uncached pixel, or compare with the CDN’s own AI crawler analytics.
Are AI browsers like Atlas or Comet counted as bots?
Usually not. They’re Chromium browsers running your scripts like any visitor, and Atlas was found to send the same user agent as Chrome, so analytics count those sessions as people. Signed requests (as ChatGPT’s cloud browser sends) and WebMCP tool calls are the clearest signs so far that an agent was involved.
Is it legal to log AI bot visits?
Logging requests from automated agents is standard server practice. Keep it to known bots, avoid storing data about ordinary visitors, and mention it in your privacy policy; take advice for your own jurisdiction.
Sources
- Overview of OpenAI crawlers, OpenAI
- Help ensure cloud browser access to your site, OpenAI
- Publishers and developers FAQ, OpenAI Help Center
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic
- Perplexity crawlers, Perplexity
- Google’s user-triggered fetchers, Google Search Central
- Google’s common crawlers, Google Search Central
- Verifying Google’s crawlers and fetchers, Google Search Central
- Web Bot Auth, Cloudflare Docs
- Forget IPs: using cryptography to verify bot and agent traffic, Cloudflare Blog
- The crawl-to-click gap: Cloudflare data on AI bots, training, and referrals, Cloudflare Blog
- The crawl before the fall… of referrals, Cloudflare Blog
- Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives, Cloudflare Blog
- Analyze AI traffic, Cloudflare AI Crawl Control docs
- How the Cache works, Cloudflare Workers docs
- [GA4] Default channel group, Google Analytics Help
- [GA4] Known bot-traffic exclusion, Google Analytics Help
Further reading
- A deeper look at AI crawlers: breaking down traffic by purpose and industry, Cloudflare Blog
- Introducing ChatGPT Atlas, Simon Willison
See every AI agent that visits, and what it did next.
Hal logs AI visits on the server and gives agents a way to send you leads. Free during early access.
Or run the free AI-readiness scan first.