TriRoute unifies attention, expert, and cache routing in single controller
A new preprint introduces TriRoute, a unified routing mechanism that jointly decides attention mode, expert selection, and cache precision for every token at every layer, outperforming independent MoE, MoD, and KV-quantization at matched compute.
TriRoute is a lightweight controller that jointly manages three conditional-computation axes—attention resolution, expert selection, and KV-cache bit-width—in a single coordinated policy per token per layer. Leading sparse techniques act in isolation: Mixture-of-Experts sparsifies the feed-forward network, Mixture-of-Depths skips entire transformer blocks, and KV-cache quantization compresses attention memory. Yet a token rare enough to need full attention may also require high-precision caching regardless of which expert processes it. TriRoute addresses that coupling by training a shared controller end-to-end via Gumbel-Softmax relaxation for categorical decisions, top-k gating for experts, and a Lagrangian budget constraint that turns average compute and memory cost into a tunable knob.
The authors identify a "cross-axis routing-collapse cascade" in naive joint training, where collapse on one axis propagates to the others, and counter it with per-axis normalization and a coupling-aware balancing loss. On decoder-only models from 160M to 1.3B parameters at compute-optimal token counts, TriRoute Pareto-dominates the best independent MoD+MoE+KV-quantization combination at matched inference FLOPs and memory while better preserving tail-case robustness on rare entities, code, and arithmetic. Post-hoc analysis shows the controller allocates full attention and high-precision cache to sentence-initial positions, rare subwords, and named entities, routing function words cheaply. The preprint appeared on arXiv on July 9, 2026.
