Publishers have spent the last few years playing a game of whack-a-mole with AI crawlers. Each company came with its own crawler name and its own opt-out, and a publisher that wanted its work out of training data had to say so to each company separately: GPTBot here, Google-Extended there, ClaudeBot somewhere else, with the list growing every time a new lab appeared. The announcement on 30 October 2026 promises to end that chore. OpenAI, Google, Microsoft and Anthropic say they will honor one shared signal, a policy token inside robots.txt, so that a publisher can set its AI-training preference once and have it apply across all four companies.
That is a genuinely useful simplification — and a narrower win than the headline suggests. What the four companies have agreed to is a training opt-out, not a scraping opt-out, and the difference matters more than the agreement itself.
The one-line fix
The token is called __usage_ai-training, and it lives in robots.txt, the plain-text file that tells automated crawlers what they may do on a site. A site that wants out adds two lines: a User-Agent line naming the token, and a Disallow line for the whole site. Publishers that want finer control can carve out exceptions for specific paths instead of blocking everything.
The token comes from an individual Internet-Draft written by Kevin Kelley, a technical lead for standards at Anthropic, and put forward through the Internet Engineering Task Force’s AI Preferences working group. That provenance is worth stating plainly, because it sets the token’s legal and institutional weight: the draft is not endorsed by the IETF, it has no formal standing in the standards process, and it expires in May 2027 unless it is updated. What publishers are being offered is a shared convention among four companies, not a rule of the web.
What the opt-out means for AI crawlers
Here is the fine print, and it is doing most of the work. The token covers AI training only. It does not stop a crawler from visiting a page, it does not remove a page from search results, and it does not keep a page out of AI-written answers that quote it. A publisher that adds the token has refused one future use of its content, not all of them.
That distinction bites because the uses are not equal. Training is how tomorrow’s models are built; answers are how today’s readers encounter content. A site can now decline to feed the next model while still being quoted, summarized and answered from by the current one. Publishers who add the token expecting to disappear from AI products will be disappointed, and publishers who read the announcement as an exit from scraping will have misread it. The token is a dial for one use, not a door out of the building.
Google’s own handling of the token underlines the point. If a site uses the shared token without naming Google-Extended, Google will treat that as blocking Google-Extended. But a site that specifically allows Google-Extended stays eligible to appear, and get clicks, in Gemini app answers. Training and answers are being treated as separate choices, and publishers are being asked to choose them separately.
Four timelines, one announcement
The second piece of fine print is timing. The four companies did not agree to move together; they agreed to move at four different speeds. Anthropic has already adopted the signal. OpenAI says it will implement it in the coming months, without committing to a specific date. Microsoft starts honoring it on 1 February 2027. Google says it aims to support it in 2027, again without a specific date.
For now, the shared token is partly promise and partly reality: protection switches on company by company as each stated implementation arrives. The announcement is best read as a timetable with a headline attached, not as a switch that has already been thrown.
A standard that is not one yet
The third piece of fine print is enforcement, or rather its absence. The token works only if crawlers choose to respect it. Nothing in the draft compels a company that has not signed up, and nothing stops a smaller lab or a non-compliant scraper from ignoring it entirely. The four backers are making a promise, not submitting to a rule, and the draft’s own status — an individual proposal with no formal standing — matches that modesty.
The guest list has a conspicuous gap, too. Meta signaled support for a common signal but did not say it will honor the opt-out itself. That is not a footnote; it is a hole in the middle of the shared switch. A publisher that opts out through the token has expressed a preference to four companies that say they will honor it once implemented, not to the web.
None of this makes the token worthless. A voluntary shared signal from the four participating labs is more useful than no signal, and the per-crawler patchwork it replaces was genuinely burdensome. But its value depends entirely on adoption, and adoption is exactly what is still incomplete.
Microsoft’s tell
If you want to know where this fight is really heading, look at what Microsoft added around the token. The company is introducing a page-level training setting and a separate AI Grounding setting, which decides whether a page can feed live AI answers in Bing, Copilot and Microsoft 365. A third option, DisallowExtended, is also named. These settings replace Bing’s old NoArchive and NoCache tags, and sites that rely on those tags have until 1 March 2027 to switch.
Microsoft is also careful to separate AI uses from search: a site’s training or grounding preferences will not remove it from Bing search results. And it is blunt about the bargain, cautioning in its own words that neither choice guarantees inclusion, traffic or payment. That is the honest version of the whole announcement. Publishers are being given controls, not guarantees.
That segmentation may be the real shift. Microsoft’s settings treat training, grounding and search as separate choices, each with its own switch and timeline. Whether the wider market follows that split, with separate negotiations and trade-offs for each use, is an open question the announcement does not settle.
What publishers should actually do
So should a publisher add the token? The practical case is straightforward: adding two lines to robots.txt puts a publisher on record for the day each company’s support switches on. But it should be added with eyes open. It refuses training, not scraping. It expresses a preference to four companies, not to the web. It takes effect on four timelines, not one. And sites using Bing’s legacy tags need to migrate by March 2027 or lose the protection those tags provided.
The shared token is also worth comparing with the alternatives publishers have been reaching for. The Really Simple Licensing standard lets publishers charge crawlers for access rather than merely refusing them, and Cloudflare now blocks AI crawlers by default for its customers. Refusal, payment and blocking are three different answers to the same problem, and the token only offers the first. Publishers who want compensation, or who want crawlers kept out entirely, will still need something else.
The four labs deserve credit for agreeing at all. But credit is not coverage. The Verge headline promises an opt-out from scraping; the document delivers an opt-out from training, on a voluntary basis, on a staggered schedule, from four companies, under a draft that expires in May 2027. That is progress, and it is precisely as large as it is — no larger. Publishers should take the win, add the lines, and then keep negotiating, because the rest of the fight is still ahead of them.
