AstraFlow cuts multi-policy LLM reinforcement learning time by 2.7×

AstraFlow is a reinforcement learning system that replaces trainer-centered control architectures with autonomous component abstractions. Developed by Haizhong Zheng, Yizhuo Di, Jiahui Wang, Shuowei Jin, Xueshen Liu, and Yongji Wu, the system decouples rollout services, dataflow management, and training into independent modules, enabling native support for multi-policy collaborative training and efficient use of elastic, heterogeneous, and cross-region compute resources. As reinforcement learning becomes standard practice for improving reasoning, coding, and tool-use capabilities in large language models, the infrastructure required to train agentic systems at scale remains prohibitively expensive and inflexible.

Existing LLM RL systems typically require dedicated engineering for each new capability extension — a burden stemming from tightly coupled control logic and the absence of principled abstractions for system components. AstraFlow's dataflow-oriented design treats rollout, data management, and training as autonomous services communicating through well-defined interfaces, allowing the same system to handle diverse workloads without system-level code changes. Evaluation across math, code, search, and AgentBench workloads showed that AstraFlow handles multi-policy training, elastic scaling, heterogeneous cross-region execution, and composable data algorithms without modification. In multi-policy collaborative training scenarios, AstraFlow delivered comparable or better accuracy than existing RL systems while reducing training time by 2.7×, a significant gain in a field where training runs consume weeks of GPU time and cost tens of thousands of dollars. The preprint was published on May 19, 2026.