ReCoLoRA cuts catastrophic forgetting in continual LLM fine-tuning
A new arXiv preprint introduces recursive consolidation that re-decomposes weights before each task, outperforming LoRA and PiSSA on continual GLUE benchmarks while training fewer parameters.
ReCoLoRA is a spectrum-aware framework for continual fine-tuning of large language models that solves a core problem in parameter-efficient adaptation: when stacking low-rank updates across multiple tasks, each new task tends to overwrite what the model learned from the previous ones. Researchers posted the preprint to arXiv on July 10, 2026.
The method re-decomposes the effective weight matrix before each new task rather than stacking low-rank updates on the original frozen weight. Instead of initializing each new adapter from the pretrained weight—the standard LoRA approach—ReCoLoRA decomposes the current effective weight (which already reflects all prior tasks) into a frozen residual, a slowly updated principal component, and a fresh adapter. Per-layer effective ranks are chosen by an elbow criterion applied to a randomized SVD, and the principal subspace is adapted before residual capacity is opened. This means every task starts from a model that has already absorbed its predecessors, reducing interference and catastrophic forgetting.
Across a six-task GLUE sequence on four 7-8B parameter backbones, ReCoLoRA achieved the best final average score on three of the four models compared to rank-swept LoRA, PiSSA, AdaLoRA, and DoRA baselines, while training fewer parameters overall. An oracle-routed task-bank variant—where each task gets its own isolated adapter—serves as the performance ceiling under full task isolation, confirming that recursive consolidation trades off some per-task capacity for improved continual learning.
What stands out
- 01Fewer parameters trained. ReCoLoRA achieves higher scores than baselines while training fewer parameters, making continual fine-tuning cheaper to run.
- 02Three-of-four backbone wins. On the six-task continual GLUE sequence, ReCoLoRA posted the highest final average score on three of the four 7-8B backbones tested.
- 03Recursive decomposition. Before each new task, the method re-decomposes the current effective weight rather than the original pretrained weight, so the model carries forward what it learned.
