Attractor Models cut training memory while boosting reasoning accuracy 46.6% over standard Transformers
Researchers introduce Attractor Models, a recurrent architecture that iteratively refines outputs by solving for fixed points via implicit differentiation, cutting training memory while outperforming Transformers and looped models on language and reasoning tasks.
Attractor Models, a new recurrent architecture from researchers Jacob Fein-Ashley and Paria Rashidinejad, address the instability and fixed-depth constraints of looped Transformers by solving for fixed points during training. The approach splits computation into a backbone module that proposes output embeddings and an attractor module that refines them adaptively until convergence, with gradients computed through implicit differentiation. Training memory stays constant regardless of effective depth, and the model chooses iteration count dynamically based on convergence rather than a hard-coded recurrence limit. The preprint reports perplexity improvements up to 46.6 percent and downstream accuracy gains up to 19.7 percent over standard Transformers across model sizes, while reducing training cost. A 770M-parameter Attractor Model outperforms a 1.3B Transformer trained on twice the token count.
On reasoning benchmarks, a 27M-parameter Attractor Model trained on roughly 1,000 examples hits 91.4 percent accuracy on Sudoku-Extreme and 93.1 percent on Maze-Hard—tasks where Claude and GPT o3 fail completely and specialized recursive reasoners collapse at scale. The paper also describes "equilibrium internalization," a phenomenon in which fixed-point training moves the backbone's initial output close enough to equilibrium that the attractor solver can be removed at inference with minimal accuracy loss. The preprint was posted to arXiv on May 13, 2026.