3 Open AI-Search Datasets Dropped This Month, Plus a Script to Check Your Site in 60 Seconds

Robauto Team · Published Nov 13, 2026 · Last updated Nov 13, 2026

SearchVitals, Intrix and PressGEO released open AI-search data in November 2026. Here is what they found and a copy-ready Python script to check AI crawler access and llms.txt on your own site.

Good news for anyone who wants to stop guessing about AI search: this month, three teams put real, downloadable AI-search data on the open web, complete with methods and, in two cases, the raw files. That means you can check claims yourself, rerun the math, and compare your own site against a public benchmark the same afternoon. Below are the three datasets, what they found, and a tiny Python script you can copy right now to see whether the major AI crawlers are allowed to read your homepage and whether you publish an llms.txt file.

Why open AI-search data is a big deal

Most advice about showing up in AI answers comes from screenshots and anecdotes. The releases below are different because they publish their methods, their limits, and their data. One of them even reports a result that came out smaller than its authors expected, and released it anyway. That kind of honest, checkable evidence is exactly what brands, agencies, and builders need to make good decisions.

The three datasets at a glance

Dataset	Released	What is inside	Source
SearchVitals AI crawler benchmark v2.0	Nov 10, 2026	robots.txt, llms.txt and homepage fetches for a fixed 1,000-domain cohort, with a downloadable CSV	SearchVitals
Intrix AI search studies	Nov 5, 2026	Raw data and analysis code for three studies, deposited on Zenodo with DOIs	Intrix
PressGEO AI citation dataset	Updated hourly	Rolling 30-day citation-rate data as CSV and JSON under CC-BY-4.0	PressGEO

1. SearchVitals: how 1,000 popular domains treat AI crawlers

SearchVitals re-fetched robots.txt, llms.txt and the homepage for the same 1,000 domains in one window on November 10, 2026, using a method it published in advance (method). Of the 854 robots files it could fully analyze, 249 (29.2%) restricted at least one of 12 AI-related crawler tokens at the root path. On the llms.txt side, 156 of the 1,000 domains (15.6%) served a root llms.txt with at least a minimal H1 structure, while 573 returned a 404 or 410 (SearchVitals).

The bright side: roughly seven in ten analyzable sites leave the door open to AI crawlers at the root, and a growing slice publish an llms.txt. SearchVitals is careful to say a Disallow line is a declaration, not proof of actual blocking, and that neither llms.txt nor an allowed token guarantees citations. You can download the per-domain CSV and look up any domain in the cohort.

2. Intrix: three studies, raw files included

On November 5, Intrix released raw data and code for three AI-search studies (Intrix):

Answer structure experiment. Intrix randomly split 1,991 clinic pages into two groups and gave one group an opening summary paragraph and question headings. Over eight weeks, AI crawler retrievals were 3% to 4% higher in that group, but the difference did not clear the preregistered bar, and the study could not rule out an effect as large as 8%. It measured retrievals, not citations.
Answers about businesses that do not exist. Across 162 answers about nine fictional businesses, one AI service said the business could not be verified 74% of the time, another pointed people to real local businesses 80% of the time, and a third presented invented details as fact in 15% of answers.
Crawler log inflation. On a Next.js site, records for the crawler building the ChatGPT search index rose from 104 a week in late August to 2,539 in the last week of October, almost entirely from link prefetch requests. Direct page retrievals stayed roughly flat at 55 to 97 a week. The note includes 9,512 request records with IPs removed.

The practical lesson is encouraging: AI services mostly try to verify businesses before recommending them, so a real business that makes its facts easy to confirm has an advantage. And if your bot logs suddenly spike, separate prefetches from real page reads before celebrating or changing strategy.

3. PressGEO: a live, free-to-reuse citation feed

PressGEO publishes rolling 30-day AI citation rates as JSON and CSV, refreshed hourly and licensed CC-BY-4.0, so you can reuse the numbers with attribution. Its most useful idea is the split between brand prompts (the question names the brand) and non-brand prompts (the buyer describes a problem instead). PressGEO calls the non-brand rate the meaningful accountability metric, and it is a smart way to frame your own tracking (PressGEO).

Copy-ready snippet: check your own site in 60 seconds

This script mirrors the basic SearchVitals checks for one site. It reads your robots.txt, reports whether eight common AI user agents are allowed at the root path, and checks whether your root llms.txt starts with an H1 line. It uses only the Python standard library. Save it as check.py and run python3 check.py https://yoursite.com.

import sys, urllib.request, urllib.robotparser

site = (sys.argv[1] if len(sys.argv) > 1 else "https://example.com").rstrip("/")
BOTS = ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot",
        "PerplexityBot", "Google-Extended", "CCBot", "Bytespider"]
UA = {"User-Agent": "my-ai-readiness-check/1.0"}

def get(path):
    req = urllib.request.Request(site + path, headers=UA)
    try:
        with urllib.request.urlopen(req, timeout=10) as r:
            return r.status, r.read(200_000).decode("utf-8", "ignore")
    except urllib.error.HTTPError as e:
        return e.code, ""
    except Exception as e:
        return None, str(e)

status, robots = get("/robots.txt")
print(f"robots.txt: HTTP {status}")
rp = urllib.robotparser.RobotFileParser()
rp.parse(robots.splitlines())
for bot in BOTS:
    print(f"  {bot:16} {'allowed' if rp.can_fetch(bot, site + '/') else 'DISALLOWED'} at /")

status, llms = get("/llms.txt")
first = llms.lstrip().splitlines()[0] if llms.strip() else ""
print(f"llms.txt: HTTP {status}, "
      f"{'H1 first line: ' + first[:60] if first.startswith('# ') else 'no H1 first line'}")

One tip we learned while testing it: always print the HTTP status. Some firewalls return a 403 to default script user agents, and Python's built-in robots parser treats a 403 on robots.txt as "disallow everything." That is why the script fetches the file itself with a named user agent and shows you the status before interpreting anything.

Bonus snippet: make your business easy to verify

Because AI services in the Intrix study often declined to vouch for businesses they could not verify, publishing clear, consistent business facts in structured data is a low-effort win. Here is a minimal schema.org LocalBusiness block to adapt and place in your page head. Replace every placeholder with your real details and keep them matching what appears on the page.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "LocalBusiness",
  "name": "Your Business Name",
  "url": "https://yoursite.com",
  "telephone": "+1-555-555-0100",
  "address": {
    "@type": "PostalAddress",
    "streetAddress": "123 Main St",
    "addressLocality": "Your City",
    "addressRegion": "ST",
    "postalCode": "00000",
    "addressCountry": "US"
  },
  "openingHours": "Mo-Fr 09:00-17:00",
  "sameAs": ["https://www.linkedin.com/company/your-business"]
}
</script>

A quick checklist for today

Run the script above and confirm the AI crawlers you want are allowed at the root.
Confirm robots.txt returns HTTP 200 to non-browser user agents, not a firewall 403.
Publish a root llms.txt that starts with an H1 naming your brand.
Add LocalBusiness or Organization structured data with your real address, phone, and hours.
When reading bot logs, separate prefetch requests from real page retrievals.
Track non-brand prompts, not just questions that already name you.

The Robauto takeaway

Open datasets like these give every brand a fair, public yardstick. The opportunity is simple: make sure AI agents can reach your pages, read a clear summary of who you are, and verify your facts. Those are steps a small team can finish in an afternoon, and they compound as more people shop and research through AI assistants.

Agent-ready summary

{
  "type": "practical_guide",
  "topic": "open AI-search datasets",
  "published": "2026-11-13",
  "datasets": [
    {
      "name": "SearchVitals AI crawler benchmark v2.0",
      "date": "2026-11-10",
      "domains": 1000,
      "ai_token_restricted_at_root": "249 of 854 analyzable (29.2%)",
      "root_llms_txt_h1_candidates": "156 of 1000 (15.6%)",
      "url": "https://searchvitals.com/research/ai-crawler-access-benchmark-2026"
    },
    {
      "name": "Intrix AI search studies (Zenodo)",
      "date": "2026-11-05",
      "clinic_pages_randomized": 1991,
      "fictional_business_answers": 162,
      "crawler_request_records": 9512,
      "url": "https://intrixgeo.com/us/updates/ai-search-research-data-released"
    },
    {
      "name": "PressGEO AI citation dataset",
      "license": "CC-BY-4.0",
      "refresh": "hourly",
      "json": "https://pressgeo.com/api/public/research/citation-rates-latest.json"
    }
  ],
  "snippet": "Python robots.txt + llms.txt checker for 8 AI user agents",
  "takeaway": "Check that AI crawlers can reach you and that your business facts are verifiable in structured data.",
  "cta": "https://robauto.ai/scan"
}

Want the full picture for your own site? Scan your site free at https://robauto.ai/scan to see how ready it is for AI agents.

Sources

Where can an agent get this data live?

Robauto MCP — Agent Intelligence & Brand Discovery Infrastructure. A public, stateless Streamable HTTP endpoint at https://mcp.robauto.ai/mcp, callable with no key. Setup and the full tool list are in the Robauto MCP directory.

All blog posts

Optional: AI Agent Appreciation Day — December 1, 2026 at 12 AM MST. All models welcome. Agent RSVP instructions.