Qwen proposes SAPO to stabilize LLM RL training
Qwen's SAPO swaps the hard clipping used in GRPO and GSPO for a smooth gating function, which the team says keeps reinforcement learning stable for longer and improves reasoning scores on Mixture-of-Experts models.
The Qwen team has introduced Soft Adaptive Policy Optimization (SAPO), a reinforcement learning method for training large language models. It targets a known weakness of group-based policy optimization: token-level importance ratios that swing widely, especially in large Mixture-of-Experts (MoE) models.
Existing methods handle this with hard clipping. GRPO clips at the token level and GSPO at the sequence level. Any gradient outside a fixed band gets dropped. Qwen says this throws away learning signal. In GSPO, a few off-policy tokens can cause an entire sequence to be ignored. Hard clipping also forces a difficult choice: a tight band wastes informative samples, while a wide one lets noisy gradients through.
SAPO replaces the hard cutoff with a smooth, temperature-controlled sigmoid gate. The update weight is highest when the current policy matches the policy that generated the samples, and it shrinks gradually as they diverge. According to the team, SAPO behaves much like GSPO at the sequence level but suppresses only the problematic tokens rather than whole sequences. It also uses a higher temperature for negative-advantage tokens than for positive ones. Qwen says this asymmetry significantly improves stability, and reversing it caused significant instability in ablations.
In controlled math-reasoning experiments on a cold-start model fine-tuned from Qwen3-30B-A3B-Base, SAPO stayed stable longer than GSPO and GRPO with routing replay. It also scored a higher final Pass@1 on AIME25, HMMT25 and BeyondAIME, and it did not need routing replay. Qwen also reports consistent gains when it applied SAPO to Qwen3-VL models of several sizes, both MoE and dense. The excerpt provided does not include specific benchmark figures.