OpenAI to honor shared AI-training opt-out signal
OpenAI says it will respect a common robots.txt signal, developed through the IETF, that lets websites state AI-training preferences once for all participating providers, so publishers no longer need separate rules for each crawler.
OpenAI announced that it will honor a common signal in robots.txt that tells AI providers how a site's content may be used for foundational generative AI training.
The signal comes out of the Internet Engineering Task Force's (IETF) AI Preferences working group. OpenAI said it is "joining other participants in the working group" in announcing it. The company did not say which other participants are involved, and it did not say whether other AI providers have committed to honor the signal. OpenAI said it encourages them to.
Until now, website operators who wanted to limit AI training had to identify each provider's crawlers and write separate rules for every one. A shared signal lets an operator state one preference that applies to every participating provider.
Participation is voluntary, and the signal is not live yet. OpenAI said it aims to implement it "in the coming months" and will publish details of its implementation at that point. The announcement does not give the signal's syntax, a timeline beyond that window, or how OpenAI will handle conflicts between the new signal and existing rules.
OpenAI said the shared signal will not replace its current robots.txt crawler controls, which will "continue to apply and be respected." The company added that it will keep adapting its practices as the IETF discussions continue.