Distilled RL transfers knowledge across model families without unconditional imitation
A new preprint proposes Distilled Reinforcement Learning, which combines fine-grained teacher guidance with RL objectives to transfer knowledge across model families more effectively than standard RL or on-policy distillation.
Distilled Reinforcement Learning (Distilled RL) is a post-training method that integrates teacher model supervision directly into the reinforcement learning objective, addressing two persistent limitations in model training. Standard RL relies on coarse-grained outcome supervision, making credit assignment difficult and limiting knowledge acquisition. On-policy distillation (OPD) unconditionally matches teacher logits through KL divergence, which restricts effective transfer to within-family model pairs. Distilled RL selectively transfers new knowledge without unconditional imitation, using three components—reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization—to provide fine-grained guidance during training.
Experiments across both within-family and cross-family distillation settings show Distilled RL substantially outperforming standard RL and OPD on pass@1 and pass@k metrics. A case study demonstrates the method can transfer previously unavailable knowledge from a teacher to a student model—a capability OPD struggles with when teacher and student diverge significantly. The key differentiator is cross-family transfer: OPD typically fails when the teacher is too dissimilar, while Distilled RL's selective transfer mechanism avoids ineffective guidance from distant teachers. This opens the possibility of training smaller models on knowledge from larger or differently-architected teachers, a practical constraint that has limited distillation in production settings.
Posted to arXiv on July 21, the preprint leaves several questions unanswered. The authors do not specify which base models were used in experiments, nor do they report wall-clock training times or compute budgets—details essential for practitioners evaluating whether the method is feasible at scale. Code is available at https://github.com/597358816/Distilled-RL. The next test will be whether geometric normalization holds up under longer context lengths and whether the method scales to frontier-sized models without degradation.
