ShortOPD cuts pruned LLM recovery time by 75% while raising generation quality 9×

ShortOPD is a training method from researchers at the Chinese Academy of Sciences that recovers generation quality in pruned large language models by focusing compute on the parts of each output that matter. Posted to arXiv this week, the paper shows that structured pruning—removing entire layers or attention heads to shrink a model—preserves enough signal that useful generations still exist in the sampling distribution, but greedy decoding collapses into repetitive loops. Standard recovery recipes waste training time on those loops; ShortOPD stops each rollout as soon as the compressed model starts repeating itself, treats the surviving prefix as the effective training length, and schedules future rollouts accordingly.

The method works by running on-policy distillation: the pruned model generates text, and a frozen copy of the original unpruned model scores each token as a teaching signal. Long rollouts under standard OPD spend most of their budget on low-information repetitive suffixes, which delays the loss curve. ShortOPD detects when the teacher confirms a suffix is repetitive, truncates the rollout there, and reallocates the saved compute to the effective lengths the policy can currently handle. Across math, code, and open-ended generation benchmarks, this schedule raises the compressed model's score to roughly 9× its unrecovered baseline and 1.6–4.4× what supervised fine-tuning or standard knowledge distillation achieve.

The efficiency gain is concrete: ShortOPD matches a fixed 8,192-token rollout horizon within two points using 8.5 hours of training instead of 35.9 hours, and 71% fewer rollout tokens. The authors tested the approach on models that had been structurally pruned—meaning entire components removed, not just weight magnitudes zeroed—and found that the gap between multiple-choice perplexity (where pruning looks fine) and free-form generation quality (where it collapses) closes under the short-to-long schedule.

The paper positions ShortOPD as a step toward making structured pruning viable for deployment, where generation quality matters more than perplexity on held-out test sets. The next question is whether the same schedule generalizes to other compression methods—quantization-aware training, mixture-of-depths routing, or speculative decoding—that also leave useful signal in a degraded sampling distribution. If the pattern holds, the broader takeaway is that recovery budgets should follow the policy's current failure modes, not a fixed horizon.