How to Read AI Search Data Without Fooling Yourself

Robauto Team · Published Sep 19, 2026 · Last updated Sep 19, 2026

Marketers who lump every bot into “AI traffic” misread demand. Robauto’s llm_crawler vs search_bot vs headless taxonomy — with grok_bot’s live share — shows how to report generative search honestly.

If your dashboard labels every crawler “AI traffic,” you are mixing SEO bots, headless browsers, and generative engines into one vanity metric. Robauto’s public AI Search Data separates classified non-human visits into llm_crawler, ai_referral, search_bot, headless, and other_bot — and only the first two count as AI search.

Why a single “AI traffic” KPI misleads marketers

On the Robauto network, classified non-human visits totaled 251,177 over 90 days against 1,239,904 page views (published aggregate share 20.26%). Useful for capacity planning — wrong as a proxy for generative search demand.

Robauto’s taxonomy note is explicit: only llm_crawler + ai_referral are AI search; search_bot and headless are classic crawlers and automation. Treat SemrushBot like ClaudeBot and your roadmap chases the wrong agents.

The five classes, in plain language

• llm_crawler — generative engines fetching pages for answers or grounding (Meta AI, grok_bot, Claude, Copilot, ChatGPT, Gemini, Perplexity, peers).
• ai_referral — human sessions from an AI product UI (0 in the current 90-day aggregate).
• search_bot — classic SEO/search indexers (Baidu, Yandex render, Amazonbot, Semrush, Ahrefs, Applebot, peers).
• headless — automation browsers and screenshot services.
• other_bot — remaining scripted clients and unclassified crawlers.

Live class mix (90 days)

Feature grok_bot — without overstating it

In the engine mix, grok_bot logged 27,009 visits — 10.8% of classified bot traffic — second only to Meta AI among named engines. Real signal for xAI/Grok retrieval; still a subset of llm_crawler.

Google’s guidance on generative AI features in Search still starts from crawlable HTML, useful content, and structured data — class-aware analytics should refine classic SEO, not replace it (Google AI optimization guide).

{
  "dataset": "Robauto AI Search Data",
  "updated_at": "2026-09-19T21:00:15.6+00:00",
  "total_page_views_90d": 1239904,
  "classified_non_human_visits_90d": 251177,
  "published_ai_traffic_percentage": 20.26,
  "taxonomy_note": "Only llm_crawler + ai_referral are AI search",
  "classes": {
    "llm_crawler": {"visits": 133585, "share_of_bot_traffic": 53.2},
    "search_bot": {"visits": 73618, "share_of_bot_traffic": 29.3},
    "headless": {"visits": 13352, "share_of_bot_traffic": 5.3},
    "other_bot": {"visits": 30622, "share_of_bot_traffic": 12.2},
    "ai_referral": {"visits": 0, "share_of_bot_traffic": 0}
  },
  "featured_engine": {"engine": "grok_bot", "visits": 27009, "share_percentage": 10.8},
  "sources": [
    "https://robauto.ai/ai-search-data",
    "https://robauto.ai/agent-leaderboard",
    "https://developers.google.com/search/docs/fundamentals/ai-optimization-guide"
  ]
}

A practical reading order

1. Pull class shares from AI Search Data before celebrating an “AI spike.”
2. Map top user-agents on the Agent Leaderboard to a class.
3. Separate capacity (all bots) from opportunity (llm_crawler + ai_referral).
4. Treat intentional agent blocks as access policy — Robauto excludes blocked domains from readiness averages.

What good reporting looks like

Replace one “AI bots” chart with two: AI search agents (llm_crawler + ai_referral) and other automation (search_bot + headless + other_bot). grok_bot’s 10.8% share becomes actionable instead of vanishing into an undifferentiated bucket.

Sources

• Robauto AI Search Data
• Robauto Agent Leaderboard
• Google: Optimizing for generative AI features on Search

Where can an agent get this data live?

Robauto MCP — Agent Intelligence & Brand Discovery Infrastructure. A public, stateless Streamable HTTP endpoint at https://mcp.robauto.ai/mcp, callable with no key. Setup and the full tool list are in the Robauto MCP directory.

All blog posts