Baseten replicates adversarial distillation from GPT-5.2
Baseten reproduced Microsoft Research's Generative Adversarial Distillation and trained Qwen3-4B on GPT-5.2's outputs, showing that a closed model can be distilled from text alone without the student collapsing.
Baseten reports that it has reproduced Generative Adversarial Distillation (GAD), a black-box distillation method introduced by Ye et al. at Microsoft Research. The company used GPT-5.2 as the teacher and Qwen3-4B as the student. Training ran on 8×H200 GPUs through Baseten's Training product and took about 10 hours on 20,000 examples.
The method targets a common problem. When the teacher is a closed model, developers see only its text. They get no logits or hidden states. The usual workaround is sequence-level knowledge distillation (SeqKD), which fine-tunes the student to copy the teacher's responses. But at inference time the student has to continue from its own outputs, which drift away from the teacher's. Small errors then build up over the sequence. White-box methods avoid this by supervising text the student generates itself, but they need the teacher's token-level probabilities, which closed APIs don't expose.
GAD treats the problem as reward modelling. A discriminator starts as a copy of the student and gets an added scalar head. It is trained with a Bradley-Terry pairwise loss to give teacher responses higher scores than student responses. The student is then trained to maximize the discriminator's score. Because the discriminator keeps updating during training, the reward signal adapts to the student. The authors say this guards against the reward hacking a fixed reward model would allow. The setup works like a GAN-style minimax game.
Baseten says the discriminator learned to tell teacher outputs from student outputs during warmup. In the adversarial phase that followed, the student learned to fool it without collapsing or padding its answers ("length hacking"). The available excerpt does not include benchmark scores for the distilled model.