HOLA cuts linear-attention perplexity 16.1% with surprise-gated token cache

Hippocampal Linear Attention (HOLA), introduced by Wanyun Cui, pairs a Gated DeltaNet recurrent backbone with a 64-token episodic cache per layer. Instead of a sliding window, HOLA selectively stores the most informative historical tokens using a parameter-free "surprise" metric computed directly from the delta-rule update residual. A separate, sharper read mechanism with scaled normalization retrieves cached tokens via near-argmax attention without destabilizing the recurrent state. The approach adds less than 0.004 percent overhead in parameters while preserving flat O(1) memory scaling at inference time.

Linear attention models promise constant memory at inference but struggle with multi-hop associative recall and long-context retrieval because they compress history with loss. HOLA addresses that by pairing the recurrent state-space backbone with a small, exact KV cache. The surprise gate decides which tokens to keep based on how much each token updates the recurrent state under the delta rule—no learned parameters, no additional training signal. On Wikitext-103, HOLA drops perplexity by 16.1 percent over base Gated DeltaNet and outperforms vanilla full-attention baselines, scaling confidently to 32k-token context with negligible memory cost. The design builds on Gated DeltaNet, the linear-attention variant from December 2024, and offers a practical path to long-context processing on edge devices without the memory explosion of full attention. The arXiv preprint (2607.02303) was posted July 5, 2026; no public code or weights are available yet.