How to Measure OAI-SearchBot

OAI-SearchBot indexes sites for ChatGPT search citations. Measure crawl coverage in logs and Hardal, separate from GPTBot training and ChatGPT Ads.

Ecem Bircan
Data Analyst
Marketing

OAI-SearchBot is OpenAI's search crawler. It exists so ChatGPT search can surface and cite websites. OpenAI says sites that opt out will not be shown in ChatGPT search answers, though they can still appear as navigational links. Full detail: Overview of OpenAI Crawlers.

Disallowing GPTBot does not remove you from ChatGPT search. They are separate tokens, and so are ChatGPT-User for live fetches and OAI-AdsBot, which only visits landing pages you submitted as ads. None of them replaces ChatGPT Ads conversion tracking.

If you want to be cited in ChatGPT search, this is the bot to measure and to allow.

How it shows up

Example UA (version will move): compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Match OAI-SearchBot. IPs: openai.com/searchbot.json. OpenAI notes robots.txt changes can take about 24 hours to apply for search.

The crawl itself produces no GA4 pageview.

What to report

  • Coverage: product, docs, and the pages you would want quoted in an answer
  • Gaps: URLs you care about with zero OAI-SearchBot hits
  • Side-by-side with GPTBot. Heavy training crawl and empty search crawl means you trained the model and still do not show up in search answers.

Human sessions that arrive from ChatGPT after a citation are a referrer problem, same caveats as ads: stripped headers, UTMs, client-side tags. First-party server-side collection still wins for those sessions. The index itself only exists in origin logs.

robots.txt

Allow search, block training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Opt out of ChatGPT search indexing:

User-agent: OAI-SearchBot
Disallow: /

ChatGPT-User is not the control for search opt-out. OpenAI says use OAI-SearchBot for that.

How Hardal measures it

Hardal names it as a searcher under OpenAI in AI Visibility. Setup and waitlist: docs and /ai-visibility. Pair with isvisible.ai for a point-in-time access check.

Ads conversions: Hardal OpenAI Ads destination and the Measurement Pixel.