One Line in robots.txt Will Soon Tell Major AI Companies: Don't Train Here

On Friday, Google said it will support a shared AI training control in robots.txt - one standard "policy token" that lets your website state its preference to every participating AI company in a single step. OpenAI, Microsoft and Anthropic are backing the same signal. If you have ever tried to opt your site out of AI training before, you know the current routine: a separate instruction for every crawler, GPTBot here, ClaudeBot there, Google-Extended somewhere else, and a new one every time a lab spins up another bot. The point of this move is to replace that sprawl with one line everyone agrees to read.

Who will honor the shared robots.txt training opt-out: Anthropic already does, OpenAI says the coming months, Google and Microsoft plan 2027, and Meta is supportive but not committed.

The standard is not a vendor invention. It comes out of the IETF, the body that maintains the internet's plumbing, where a working group called AIPREF has been drafting a shared vocabulary for exactly this. The proposal's author is a standards lead at Anthropic. The EU's AI Act code of practice already treats robots.txt as the primary way publishers express machine-readable preferences, which gives the line some legal weight in Europe on top of the industry agreement.

What it does and does not do, because the details decide whether you should care:

It covers training. This signal is about whether your pages end up inside future models. It is separate from being found: Google and Microsoft both say opting out of training does not touch your appearance in Search or Bing, and Microsoft's version splits training from grounding - the citations AI answers link back to. You can say no to the classroom and yes to the footnotes.

It is a preference, not a wall. Each company honors it voluntarily, on its own timeline. If you want a technical block today, that is the Cloudflare route - they already block AI crawlers by default unless you say otherwise. And nothing is retroactive: whatever already went into today's models stays there.

It mostly arrives next year. Anthropic already reads the signal. OpenAI says the coming months. Google and Microsoft say 2027. So today, opting out still means the per-company instructions; the one-line version is the near future, not this afternoon.

So why does a switch you may never flip matter? Because once one line exists that the major labs recognize, your terms become machine-readable. There is already a licensing standard built to attach payment to crawling, and Cloudflare has turned crawler control into a dashboard. A recognized preference is the hook those hang on. My read: the switch is not the story, the standard is - this is how "my content, my terms" stops being a legal letter and starts being a setting.

What it means for operators (Jaden's take)

First, decide your default before this lands in your host's dashboard - if the standard holds, a checkbox there is the natural place for it to surface. If you run a local or service business, staying opted in is the easy default: your website is your brochure, and the training choice does not gate the distribution you actually want. Being found in search and quoted in AI answers runs on separate controls, so decide training on its own merits. If what you sell IS content - courses, research, photography, a paid archive - learn where this setting is likely to live and treat it like pricing: my read is that the standard that says "no" today is the one most likely to say "for a fee" later. Second, if you care about this today, don't wait for 2027 - the per-company tokens and Cloudflare's block both work right now. The quiet plumbing changes are the ones that end up mattering, and this one just got the four biggest names in AI to agree on where the valve goes.