Hypernetworks outscale LoRA for train-time knowledge injection in LLMs
New arXiv preprint from Dhankhar, Baha, and Saparov shows hypernetworks exhibit steeper scaling exponents than LoRA fine-tuning when injecting factual knowledge into large language models at training time.
A preprint published on arXiv this week establishes the first empirically grounded scaling laws for hypernetwork-based knowledge injection in large language models. Hypernetworks are small neural networks that generate adapter weights for a target model — typically used at test time, but the authors demonstrate their use at training time to inject factual knowledge at scale. The team trained hypernetworks to produce fixed LoRA adapters that, when inserted into a target LLM, enable the model to answer multi-hop questions about a large corpus of facts. By decoupling the hypernetwork's injection capacity from the target model's general capability, the researchers characterized how loss, reasoning accuracy, and out-of-distribution generalization vary with hypernetwork depth, width, and target network size.
The results show hypernetwork-based injection follows predictable power law scaling along all architecture axes and achieves reliable OOD generalization at increasing scales. Critically, hypernetworks exhibit steeper scaling exponents than both LoRA fine-tuning and full fine-tuning across all OOD evaluations, suggesting they're a more efficient substrate for train-time adaptation. The team constructed MegaWikiQA, a dataset containing tens of millions of multi-hop question-answer examples across 39 domains from Wikidata5M, to validate the approach.
