Sequential training beats arbitrary-order generation in diffusion language models
Researchers discovered that forcing diffusion language models to generate tokens left-to-right during reinforcement learning produces better reasoning than allowing arbitrary token order, winning an Outstanding Paper Award at ICML 2026.
A team led by Zanlin Ni at Tsinghua University has identified what they call the "flexibility trap" in diffusion language models — the counterintuitive finding that letting these models generate tokens in any order during training actually degrades reasoning performance. The work, which earned an Outstanding Paper Award at ICML 2026, introduces JustGRPO, a training framework that constrains diffusion LLMs to standard left-to-right generation during the exploration phase of reinforcement learning while preserving bidirectional attention and parallel decoding at inference time.
The researchers traced the problem to "entropy degradation": when given freedom to choose token order, models sidestep logically demanding positions with high uncertainty. By enforcing sequential generation during RL training only, JustGRPO pushed a diffusion LLM to 89.1 percent on GSM8K and 45.1 percent on MATH-500 — competitive math and coding benchmarks — without the complex mathematical approximations or unstable RL formulations previous methods required to maintain arbitrary token order.
The preprint is available on arXiv at arxiv.org/abs/2601.15165, with code at github.com/LeapLabTHU/JustGRPO and a project page at nzl-thu.github.io/the-flexibility-trap.
