Lightweight proxy models cut LLM post-training costs while enabling cross-model signal reuse
Researchers propose Proxy-guided Update Signal Transfer, a post-training method that uses lightweight proxy models to discover optimization signals and transfer them to larger primary models, cutting computational overhead while enabling cross-model reuse.
Proxy-guided Update Signal Transfer (PUST) decouples the expensive exploration phase of LLM refinement from policy alignment, according to a preprint posted July 14, 2026. Instead of running costly reward optimization directly on a large primary model, PUST runs exploration on a small proxy model, extracts the relative improvement signal between the proxy's initial and optimized states, and transfers that directional update to the primary model. The result is a modular pipeline where optimization signals can be generated asynchronously, cached, and reused across different models.
Existing post-training methods tightly couple exploration and distribution alignment, forcing the primary model to discover high-reward behaviors on its own—a computationally expensive process that prevents signal reuse. PUST's decoupled design means a single proxy exploration run can guide multiple primary models. Crucially, signals extracted from weaker proxies can robustly improve stronger models, a weak-to-strong transfer capability that existing methods lack.
Systematic evaluations on Qwen3-family models across math and code domains show that update signals from substantially smaller proxy models reliably enhance larger primary models, demonstrating the framework's modularity and cost efficiency.
