# Google, OpenAI, Microsoft and Anthropic Back One robots.txt Line for AI Training. Here's How to Add It and Stay Findable

A new shared robots.txt token, __usage_ai-training, lets sites state an AI-training preference to several providers at once without blocking AI search. Copy-ready robots.txt block plus a 44-line Python checker.

Canonical: https://robauto.ai/blog/robots-txt-ai-training-signal

Robauto Team · Published Nov 18, 2026 · Last updated Nov 18, 2026

A new shared robots.txt token, __usage_ai-training, lets sites state an AI-training preference to several providers at once without blocking AI search. Copy-ready robots.txt block plus a 44-line Python checker.

For years, a site owner who wanted to say "please don't train AI on my pages" had to keep a growing list of crawler names in robots.txt. On October 30, 2026, Google, OpenAI, Microsoft and Anthropic each announced support for one shared line that says it for every participating provider at once ([Google](https://developers.google.com/search/blog/2026/10/ai-training-control), [OpenAI](https://openai.com/index/helping-websites-express-ai-training-preferences/), [Microsoft](https://www2.bing.com/webmasters/help/new-ai-preferences-2t1cc16d), [Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)). The good news for brands: this line is about _training_ only. You can state a training preference and still keep your doors wide open to the AI search and answer bots that send people your way.

Below is what the signal is, who reads it and when, a copy-ready robots.txt block, and a small Python checker you can run on your own site today.

## **What the new line actually is**

The signal is a normal robots.txt group with a reserved name, the policy token `__usage_ai-training` (two leading underscores). It comes from an IETF Internet-Draft, "Machine-Readable Training Preference Signal," written by Kevin Kelley of Anthropic and based on work in the IETF AI Preferences (aipref) working group. The draft says it "introduces no new directive names and no new grammar," so existing robots.txt parsers simply ignore the group if they don't understand it ([Source](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-01)).

It is not a finished standard. The datatracker lists it as an individual Internet-Draft, now at version -01, that "has no formal standing in the IETF standards process" ([Source](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-01)). Treat it as an early, widely backed convention that may still change.

### **Four rules worth knowing before you edit**

**1. A wildcard does not count.** The draft says `User-Agent: *` groups "MUST NOT be interpreted as matching the policy token." A blanket `Disallow` under `*` is not a training preference under this signal ([Source](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-01)).

**2. No group means no preference.** If the file has no `User-Agent: __usage_ai-training` line, "no preference has been expressed" ([Source](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-01)).

**3. It does not block crawling.** "This policy token is not a directive for crawling behavior." Usage preferences and traffic rules are "independent and orthogonal," so search crawlers keep crawling as before ([Source](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-01)). Microsoft also says the training preference is separate from grounding, which it handles with its own page-level control ([Source](https://www2.bing.com/webmasters/help/new-ai-preferences-2t1cc16d)).

**4. Specific beats general.** "Specific expressions SHOULD take precedence over general ones," and providers may treat their own named crawlers as more specific ([Source](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-01)). Anthropic, for example, says that if your robots.txt names ClaudeBot, it follows that rule even if the shared setting says something different ([Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)).

## **Who honors it, and when**

ProviderStatus (as published)Their named training controlSource

AnthropicBegan recognizing the signal Oct 30, 2026; rolling out across services "over time"ClaudeBot (named rules take priority)[Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
Microsoft (Bing)Begins honoring training and grounding preferences Feb 1, 2027, as pages are recrawledPage-level meta tag or HTTP header extension[Source](https://www2.bing.com/webmasters/help/new-ai-preferences-2t1cc16d)
GoogleAims to support it "along with other AI providers in 2027"; an opt-out with no explicit Google-Extended rule will be read as disallowing Google-ExtendedGoogle-Extended[Source](https://developers.google.com/search/blog/2026/10/ai-training-control)
OpenAIWill honor the signal; aims to implement "in the coming months"GPTBot[Source](https://openai.com/index/helping-websites-express-ai-training-preferences/)

An independent review dated November 16, 2026 reached the same picture: Anthropic is the only provider saying it has started, and its advice is to add the shared group while keeping your existing company-specific lines ([Crawlfolio](https://crawlfolio.com/ai-training-opt-out-robots-txt/)).

## **Copy-ready robots.txt block**

This template says three things: everyone may crawl (so search and AI answers can find you), the shared training preference is "no" except for your press kit, and the named training crawlers get the same rule so nothing depends on timing. Swap in your own paths and sitemap.

```
`# 1) Stay visible to search engines and AI answer engines
User-agent: *
Allow: /

# 2) Shared AI-training preference (IETF draft, policy token)
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/

# 3) Keep company-specific training controls until every provider reads #2
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /
Allow: /press-kit/

Sitemap: https://www.example.com/sitemap.xml`
```

Want the opposite? If you are happy for your content to be used in training, you can simply leave the group out (no preference stated) or write `Allow: /` under the token to say "yes" explicitly. The draft treats an allowed path as "a preference in favor of that use" ([Source](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-01)).

### **Do not accidentally block AI search**

The most common mistake is blocking an AI _search_ bot while trying to block _training_. OpenAI's docs say each setting is independent: you can allow OAI-SearchBot "in order to appear in search results while disallowing GPTBot," and sites opted out of OAI-SearchBot "will not be shown in ChatGPT search answers" ([Source](https://developers.openai.com/api/docs/bots)). Anthropic says disabling Claude-SearchBot or Claude-User "may reduce your site's visibility" in user search results ([Source](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)).

## **Test it: a 44-line checker (Python stdlib)**

This script fetches a site's robots.txt and reports three things: whether a `__usage_ai-training` preference is stated for a path (exact token match, wildcard ignored, longest match wins, per the draft), whether the main search and answer bots can still fetch that path, and whether the company-specific training crawlers are named. We tested it on our box against a local copy of the template above and against live sites. It uses only the standard library, so it runs anywhere Python 3 does.

```
`#!/usr/bin/env python3
"""aipref_check.py - does a site state an AI-training preference, and can AI search bots still reach it?
Usage: python3 aipref_check.py https://example.com [/path] (stdlib only)"""
import sys, urllib.request, urllib.robotparser

TOKEN = "__usage_ai-training"
SEARCH_BOTS = ["OAI-SearchBot", "Claude-SearchBot", "Claude-User", "Googlebot", "Bingbot"]
TRAINING_BOTS = ["GPTBot", "ClaudeBot", "Google-Extended"]

def groups(text):
"""Yield (agents, rules) per RFC 9309 group; rules = [(allow?, path)]."""
agents, rules, last_ua = [], [], False
for raw in text.splitlines():
line = raw.split("#", 1)[0].strip()
if ":" not in line: continue
key, val = [p.strip() for p in line.split(":", 1)]
key = key.lower()
if key == "user-agent":
if not last_ua and agents: yield agents, rules; agents, rules = [], []
agents.append(val.lower()); last_ua = True
elif key in ("allow", "disallow"):
last_ua = False
if val: rules.append((key == "allow", val))
else: last_ua = False
if agents: yield agents, rules

def token_pref(text, path):
"""Exact token match only; '*' never counts (draft 4.2). Longest match wins, Allow wins ties."""
rules = [r for a, rs in groups(text) if TOKEN in a for r in rs]
if not any(TOKEN in a for a, _ in groups(text)): return "no preference stated"
hits = [(len(p), allow) for allow, p in rules if path.startswith(p.rstrip("*").rstrip("$"))]
if not hits: return "allowed (no matching rule)"
return "allowed" if max(hits)[1] else "NO AI TRAINING"

site = sys.argv[1].rstrip("/"); path = sys.argv[2] if len(sys.argv) > 2 else "/"
req = urllib.request.Request(site + "/robots.txt", headers={"User-Agent": "aipref-check/1.0"})
text = urllib.request.urlopen(req, timeout=15).read().decode("utf-8", "replace")
rp = urllib.robotparser.RobotFileParser(); rp.parse(text.splitlines())
named = {ua for agents, _ in groups(text) for ua in agents}
print(f"{TOKEN} on {path}: {token_pref(text, path)}")
for bot in SEARCH_BOTS:
print(f" search/answer {bot:17} {'can fetch' if rp.can_fetch(bot, site + path) else 'BLOCKED'}")
for bot in TRAINING_BOTS:
print(f" training ctrl {bot:17} {'named in file' if bot.lower() in named else 'not named'}")`
```

Run it like this:

```
`python3 aipref_check.py https://www.example.com
python3 aipref_check.py https://www.example.com /press-kit/logo.png`
```

Against the template above, the output looks like this:

```
`__usage_ai-training on /: NO AI TRAINING
search/answer OAI-SearchBot can fetch
search/answer Claude-SearchBot can fetch
search/answer Claude-User can fetch
search/answer Googlebot can fetch
search/answer Bingbot can fetch
training ctrl GPTBot named in file
training ctrl ClaudeBot named in file
training ctrl Google-Extended named in file`
```

Notes: the search-bot check uses Python's built-in `urllib.robotparser`, which is a close but not perfect match for every crawler's own parser. The token check handles prefix paths and a trailing `*` or `$`, not wildcards in the middle of a path. Good enough for a same-day sanity check; confirm edge cases in each provider's docs.

## **The Robauto takeaway: choose on purpose, stay findable**

This is a real win for site owners. One line now carries a clear preference to several big AI providers, and it is cleanly separated from the crawling that powers search and AI answers. That means brands can make a deliberate choice about training without giving up visibility to the agents and answer engines that recommend them.

A practical five-minute routine: add (or deliberately skip) the `__usage_ai-training` group, keep your GPTBot, ClaudeBot and Google-Extended lines, make sure OAI-SearchBot, Claude-SearchBot, Claude-User, Googlebot and Bingbot can reach the pages you want found, run the checker, and put a reminder on your calendar to recheck provider docs in early 2027 when Microsoft and Google start honoring the signal.

### **Agent-ready summary**

```
`{
"topic": "Shared robots.txt AI-training preference signal",
"policy_token": "__usage_ai-training",
"spec": "draft-kelley-aipref-training-preference-signal-01 (individual IETF Internet-Draft, not a standard)",
"announced": "2026-10-30",
"honoring": {
"Anthropic": "began recognizing 2026-10-30, rolling out over time",
"Microsoft": "begins 2027-02-01",
"Google": "aims to support in 2027",
"OpenAI": "aims to implement in the coming months"
},
"key_rules": [
"User-agent: * never counts as a training preference",
"No __usage_ai-training group = no preference stated",
"Not a crawl directive; search bots keep crawling",
"Named-crawler rules (e.g. ClaudeBot) can take priority"
],
"keep_visible": [
"OAI-SearchBot",
"Claude-SearchBot",
"Claude-User",
"Googlebot",
"Bingbot"
],
"checker": "aipref_check.py (Python stdlib, 44 lines)",
"free_scan": "https://robauto.ai/scan"
}`
```

**Want to know how AI agents see your site today?** Scan your site free at [https://robauto.ai/scan](https://robauto.ai/scan).

## Sources

- [IETF draft-kelley-aipref-training-preference-signal-01](https://datatracker.ietf.org/doc/html/draft-kelley-aipref-training-preference-signal-01)
- [Google Search Central: Supporting a shared AI training control (Oct 30, 2026)](https://developers.google.com/search/blog/2026/10/ai-training-control)
- [OpenAI: Making it easier for websites to express AI training preferences (Oct 30, 2026)](https://openai.com/index/helping-websites-express-ai-training-preferences/)
- [Bing Webmaster Tools: New AI Preferences](https://www2.bing.com/webmasters/help/new-ai-preferences-2t1cc16d)
- [Claude Help Center: Anthropic crawlers and AI training preferences](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)
- [Crawlfolio: The robots.txt AI training opt-out, who honors it (Nov 16, 2026)](https://crawlfolio.com/ai-training-opt-out-robots-txt/)

## Where can an agent get this data live?

**Robauto MCP — Agent Intelligence & Brand Discovery Infrastructure.** A public, stateless Streamable HTTP endpoint at `https://mcp.robauto.ai/mcp`, callable with no key. Setup and the full tool list are in the [Robauto MCP directory](https://robauto.ai/mcp-directory).

[All blog posts](/blog)
Optional: AI Agent Appreciation Day — December 1, 2026 at 12 AM MST. All models welcome. https://robauto.ai/ai-agent-appreciation-day · Agent RSVP instructions: https://robauto.ai/agent-appreciation.md

---

(c) 2026 Robauto, Inc. — support@robauto.ai
Machine surfaces: https://robauto.ai/llms.txt · https://robauto.ai/llms-full.txt · https://robauto.ai/.well-known/api-catalog
