HPC-LLM fine-tunes Llama 3.1 8B to match Qwen 14B on cluster tasks
Researchers fine-tuned Llama 3.1 8B with QLoRA on HPC documentation to build a retrieval-augmented assistant for Slurm, MPI, and GPU workflows, matching larger models at lower memory cost.
HPC-LLM, a domain-adapted assistant for High-Performance Computing workflows, fine-tunes Llama 3.1 8B using QLoRA and pairs it with dense retrieval for documentation lookup. The model targets Slurm scheduling, MPI execution, GPU utilization, filesystem management, and cluster troubleshooting — tasks where general-purpose LLMs often lack the operational specifics researchers need when navigating cluster environments.
The team built a training corpus from publicly available university HPC documentation, curated operational examples, and synthetic instruction-answer pairs, totaling 9,000 to 24,000 HPC-focused examples. Experimental results show the adapted 8B checkpoint approaches the performance of Qwen 2.5 14B on HPC support tasks while running under significantly lower GPU memory requirements and inference latency. The researchers argue that lightweight domain adaptation can close the knowledge gap without the memory and latency penalties of much larger models, making the approach practical for on-premises deployment where data residency or network latency matter.
The preprint, arXiv:2605.16347, was posted on May 19, 2026.