TurnOPD cuts long-horizon agent training time with adaptive turn budgeting
New preprint from Yuhang Zhou and colleagues shows turn-aware distillation strategy that outperforms vanilla on-policy distillation on ALFWorld, WebShop, and Multi-Hop Search under equal wall-clock budgets.
TurnOPD is a training framework that makes on-policy distillation (OPD) more efficient for long-horizon language agents. The July 2026 preprint, authored by Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, and Jingjing Chen, addresses two core inefficiencies in vanilla agent OPD: wasted compute on weak tail-turn supervision and under-training of deep decision turns once early behaviors align.
On-policy distillation trains a student policy by matching a stronger teacher on the student's own rollouts. For long-horizon agentic tasks—where an agent must take dozens of actions to complete a goal—vanilla OPD often runs full trajectories even when later turns provide noisy or redundant signal. The token-level KL objective also concentrates loss on shallow tokens, leaving critical mid- and late-turn decisions under-supervised.
Adaptive budgeting mechanisms
TurnOPD introduces two complementary controllers. Adaptive rollout-depth budgeting uses probe-based turn statistics to decide when to stop a rollout early, skipping tail turns that contribute weak supervision. Progressive turn-normalized loss budgeting gradually shifts the KL weighting from token-level to turn-balanced, ensuring deeper decision points receive proportional training signal as initial behaviors converge.
Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show TurnOPD reaches higher validation accuracy than vanilla OPD under equal wall-clock training budgets. The paper reports that TurnOPD advances the accuracy–time frontier, meaning it achieves better task success rates in less wall-clock time. Ablations in the preprint isolate the contribution of each budget controller, confirming both mechanisms drive the efficiency gain.
