OpenAI chief scientist expects recursive self-improvement

In an essay titled "An Alien Mind," OpenAI's chief scientist writes: "Based on internal results, I have a strong expectation that this speed of progress could be sustained into recursive self-improvement." Systems arriving in the next few years, the essay argues, are likely to represent capability jumps "of equal or larger magnitude" and to "increasingly drive their own development."

The piece dates the lab's confidence in reasoning models to mid-2023 and an internal project called "RLSlow," which produced the first results suggesting training could be scaled to let pretrained models form their own chains of thought. Three years on, it says, reasoning models operate computers and graphical interfaces, collaborate with people and each other, carry out research projects, and are "transforming the landscape of computer security, and in that present clear new dangers."

On posture: OpenAI will keep pursuing alignment and monitoring solutions, build defensive systems, and "unilaterally withhold further scaling as needed" — but "broader interventions are required," because "no one is prepared for the consequences of a continued rapid rise in machine intelligence."

One prioritization disclosure: OpenAI believes it could make models materially better at mathematics research with added focus, but is not doing so, citing urgency around recursive self-improvement and automated alignment research. The essay splits alignment into goal alignment (does the model pursue the objective set for it, including instruction hierarchy) and value alignment (does it generalize principles under unclear, unfamiliar, or adversarial conditions).

The available text ends mid-section. The specific broader interventions being called for, and any accompanying model or incident disclosures, are not in the excerpt.