OpenAI will honor a robots.txt line for AI training preferences

OpenAI will honor a shared robots.txt signal that lets website operators say whether their pages may be used to train AI models.

OpenAI said Oct. 30, 2026, in a post titled “Making it easier for websites to express AI training preferences,” that the signal is meant to let operators communicate those preferences to participating providers.

The Signal Is an Ordinary robots.txt Group

A site operator expresses the preference with a robots.txt group whose User-Agent line carries a policy token, __usage_ai-training. A disallowed path means the operator prefers no AI training on it. An allowed path means the operator prefers training. Preferences can be set per path or for the whole site.

The draft’s own example reads User-Agent: __usage_ai-training, Disallow: /, Allow: /press-kit/. That states a preference against training for every path except the press kit.

An Unfinished IETF Draft Defines It

The signal comes from an Internet-Draft titled “Machine-Readable Training Preference Signal,” written by K. Kelley of Anthropic and dated Oct. 30, 2026. It expires May 3, 2027, and its intended status is Standards Track. The IETF said an Internet-Draft is not endorsed by the IETF and has no formal standing.

The draft adds no directive names and no new grammar. Path matching, longest-match precedence and group structure under RFC 9309 stay as they are. A parser that does not implement it ignores the group, as it would any group for an unknown user agent. A group may name several policy tokens and may mix them with ordinary product tokens.

Wildcard Groups Do Not Count

Wildcard groups cannot express a training preference. The draft says a User-Agent: * group must not be read as matching the token, so a file with no group naming __usage_ai-training has expressed no preference.

The token also sits apart from crawl rules. Consumers read the rules for their own product tokens exactly as before, and the draft says the token is “not a directive for crawling behavior.”

The draft defines AI training as using an asset to modify the learned parameters of a generative AI model. Training of models used only for non-generative tasks is excluded. The preferences that apply to an asset are those in the robots.txt in effect when the asset was retrieved.

Our View

We think the wildcard rule is the one a site operator needs to act on. A robots.txt file that carries only User-agent: * rules says nothing about training under the draft, however strict its crawl limits are.

A publisher who wants out of training therefore has to add a group naming the token. One who wants in for a section, such as a press kit, has to say so with an allow line.

The line only protects a site once each provider’s training pipeline reads it.