Blogs

How to Measure ClaudeBot

ClaudeBot is Anthropic's training crawler. Measure crawl volume and pages in logs or Hardal AI Visibility, separate from Claude-User fetches.

Ecem Bircan
Data Analyst
Marketing

ClaudeBot is Anthropic's training crawler. It reads your site so future Claude models can learn from public pages. There is no Claude ads network, and a live user fetch arrives under a different agent.

Anthropic has split the job the same way OpenAI did: training vs search vs user-initiated fetch. In logs you will also see Claude-User (someone asked Claude to open a URL) and, on newer stacks, Claude-SearchBot for in-product search indexing. Measuring "Claude traffic" as one number hides those three jobs.

Hardal's AI Visibility docs call out ClaudeBot by name and attribute its visits to Anthropic, on the trainer line. Treat fetch spikes on pricing the way we treat ChatGPT-User: live intent, not a training crawl.

How ClaudeBot shows up

Look for ClaudeBot in the user-agent. Do not collapse anthropic-ai, Claude-User, and ClaudeBot into one regex unless you also split the chart. The playbook examples use ClaudeBot next to GPTBot and PerplexityBot because those are the high-volume names marketers actually see (AI Visibility Playbook).

Client-side analytics still miss it. ClaudeBot runs no JavaScript, so there is no GA4 session.

What to report

  • ClaudeBot hits vs Claude-User hits. If trainers dwarf fetchers, Anthropic has your pages but almost nobody is asking Claude to open them.
  • Pages Anthropic reads most. Our playbook cites the ugly ratio problem: heavy crawl, almost no referred humans. That is a content and robots.txt decision, not a paid-media one.
  • Whether /docs and comparison pages are in the crawl set. If Claude is going to describe Hardal-class products, those URLs should be current.

There is no Hardal "Claude Ads" destination. Do not copy the ChatGPT Ads setup onto ClaudeBot.

robots.txt

Block training, keep fetches and search if you want to be cited:

User-agent: ClaudeBot
Disallow: /

If you also want to stay out of Claude search indexes, add the search token Anthropic documents at the time you ship the file. Tokens change. Re-read Anthropic's crawler page each quarter instead of copying a 2024 gist.

Blocking ClaudeBot does not change Googlebot rankings.

How Hardal measures it

Same pipeline as GPTBot: first-party server-side identification, with Anthropic as the platform and trainer as the category. isvisible.ai checks whether Claude can access the homepage. Continuous volume and page lists are AI Visibility (beta / waitlist).

For the after-the-report actions, use AI crawler traffic for marketers.