A shared way to opt out of AI training: what to decide for robots.txt

Short answer

On October 30, 2026, a draft was submitted to the IETF, the body that develops internet standards, for a single way to say "please do not use this site's content to train AI", instead of a separate instruction for each AI company. The same day, Google said on its official blog that it aims to support this signal, along with other AI providers, in 2027. It is still a draft, and companies will only start honouring it later. There is no need to rush a change, but it is a good moment to decide, as a business, whether your text and images may be used for training.

What was settled

The draft is titled "Machine-Readable Training Preference Signal" and expires on May 3, 2027. If you write a group named __usage_ai-training in robots.txt, the file at the root of your website, it expresses a preference that content at those paths should not be used for AI training. robots.txt was originally a way to tell crawlers such as search engines "please don't fetch this". The draft adds no new directives; it reuses the existing syntax to express a preference about use.

Until now, a site that wanted to refuse training had to look up a different name for each company, such as Google-Extended for Google, and list each one. Google says that if a site refuses training with the shared token and does not mention Google-Extended, it plans to treat that as a disallow for Google-Extended. Existing company-specific controls will keep working.

Three things to watch

"User-Agent: *" does not count

The draft states that a User-Agent: * group, which addresses every crawler, must not be read as a training preference. The reverse mistake is worse: writing Disallow: / under User-Agent: * to refuse training would also stop search engines, and your site could drop out of search results.

Writing nothing means no preference

If no group names the token, you have expressed no preference either way. Saying nothing is not the same as saying no.

Training and appearing in AI answers are different

According to Google, Google-Extended covers training future Gemini models and also "grounding", where Search content is given to the model when it answers. Sites that allow Google-Extended are eligible to appear in Gemini app answers and receive clicks from them. Google also says Google-Extended does not affect inclusion or ranking in Google Search.

What to decide for your own site now

1. Group your content: things you want widely known, like your company profile; columns you put real effort into; photos whose rights you want to protect.
2. For each group, decide whether it may be used for training and whether you want it mentioned in AI answers.
3. Add /robots.txt to the end of your site address in a browser and see what it says today.
4. Find out whether you can edit it yourselves, or whether it is changed by your web developer or through a settings screen.
5. When you change it, keep the existing company-specific lines and add the shared group alongside them.

robots.txt is a request; it cannot stop a crawler that ignores the rules. Anything you truly do not want seen needs a different measure, such as not publishing it or putting it behind a login.