LongStraw trains RL models at 2.1M tokens on eight H20 GPUs

Inference systems now routinely handle million-token contexts, but reinforcement learning post-training still clusters around 256K tokens and hopes length generalization will close the gap at deployment—a mismatch that hits hardest in AI agent workloads where observations, tool outputs, and prior decisions pile up across long trajectories.

LongStraw is an architecture-aware execution stack that brings million-token RL post-training within reach of a fixed GPU budget. Detailed in a preprint released this week, the system evaluates the shared prompt without autograd, retains only the model-specific state needed by later tokens, and replays short response branches one at a time. That design choice shrinks the live training graph at the cost of additional replay time. The authors instantiate it with Group Relative Policy Optimization (GRPO) and implement it for Qwen3.6-27B—a hybrid recurrent and full-attention model—and GLM-5.2, a compressed-attention mixture-of-experts.

On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1 million positions for groups of 2 and 8. Increasing the group size adds only 0.21 GB of peak allocated memory, while a separate stress test reaches 4.46 million positions. On 32 H20 GPUs, the team validated the end-to-end execution path for a 2.1M-token prompt across all 78 layers of GLM-5.2. The experiments establish execution capacity rather than complete training correctness—the captured prompt state is detached and some distributed forward and gradient composition paths remain incomplete—but they demonstrate that million-token RL post-training is no longer a memory wall problem on commodity clusters.