Googlebot is still the crawler that matters for Google Search. Google's AI Overviews and AI Mode also depend on that search index. It is a different agent from Google-Extended, which is the training token, and it has nothing to do with Google Ads.
Block Googlebot to "stop AI" and you leave Search. Blocking Google-Extended is the switch for Gemini training. Hardal's isvisible.ai groups Googlebot with Bingbot as search crawlers that power AI-assisted results.
How to measure it (the boring, correct stack)
- Search Console for coverage, crawl stats, and rendering. This remains the source of truth for Google's view of your site.
- Origin logs for
Googlebotin the user-agent, verified against Google's published IP and reverse-DNS guidance when you care about spoofing. - Hardal AI Visibility for the named agent under platform Google, so Googlebot is not mixed with GPTBot in one "bot" pie chart (docs).
Googlebot does execute more of the web than GPTBot, but you still should not use a marketing pixel as your Googlebot counter. Crawl stats are an ops and SEO dataset.
What to report to a marketing budget owner
- Crawl vs index vs clicks (Search Console), not "Googlebot ROAS"
- Whether AI-shaped Google features can see the same URLs as classic Search (they share the crawl)
- Side-by-side with Bingbot if Copilot / Bing is in the story
- Explicit note: ChatGPT Ads measurement is a different pipeline (hub)
robots.txt
Allow Googlebot unless you intend to disappear from Google:
User-agent: Googlebot
Allow: /
Training opt-out stays on Google-Extended. Do not copy a "block all AI bots" snippet that includes Googlebot.
How Hardal measures it
AI Visibility categorizes search-engine agents separately from LLM trainers and fetchers. Use it as the named-bot layer next to Search Console, not instead of it. Human Google Ads conversions still go through your existing Google Ads / GA4 destinations on the first-party server path.