Sber's GigaChat 3.5 Ultra: 432B hybrid model cuts memory footprint 40%, runs on MIT license
Sber released GigaChat 3.5 Ultra, a 432-billion-parameter open-weight model combining GatedDeltaNet and MLA architectures under MIT license—40% smaller than its predecessor with 4× lower KV cache and 2.14× more context capacity.
Sber open-sourced GigaChat 3.5 Ultra this week, a 432-billion-parameter model that pairs GatedDeltaNet and MLA architectures for the first time at production scale. Released under MIT license, the model cuts KV cache footprint by roughly 4× per token, fits 2.14× more context in the same memory, and delivers 20% higher throughput under load—all while shrinking to 40% of its predecessor's size. Two novel stabilization techniques—Gated Attention, which locally dampens overly strong attention signals, and GatedNorm, an explicit gating mechanism for feature-scale control—were essential to training the hybrid at this scale. Sber ran more than 1,500 experiments to finalize the recipe, trained entirely in FP8 using custom Triton and CUDA kernels with no quality loss versus bf16, and added a post-DPO online RL stage. Two MTP heads accelerate generation by up to 2.2×.
On Sber's benchmark suite spanning general reasoning, math, and code, the base checkpoint outperforms DeepSeek V3.2 Exp Base and DeepSeek V4 Flash Base on average, while the instruct variant matches DeepSeek V3.2's aggregate score at 1.5× fewer parameters. An LLM judge (MiniMax-M2.7) scored GigaChat 3.5 Ultra Instruct at a 75.9% win rate against GigaChat 3.1 Ultra and 68.7% against GPT-5. Weights and technical details—including gate implementations and the stabilization recipe—are available on HuggingFace and GitVerse, with a full writeup on Habr covering the training corpus (Sber's own LLM-parsed web crawl spanning 600+ programming languages).
