Tencent Hy3: 295B MoE model hits 256K context on standard attention

Tencent released Hy3 this week, a 295-billion-parameter mixture-of-experts model that activates roughly 21 billion parameters per token. The model supports up to 256K context length and ships under Apache 2.0, with weights available on HuggingFace alongside an FP8 quantized version. In blind tests, Hy3 outperformed GLM-5.1 while using fewer active parameters than competing models in its class.

Hy3 runs on standard grouped-query attention without sparse attention or multi-latent attention optimizations. That architectural simplicity leaves headroom for future efficiency gains—the model hasn't yet squeezed every drop of performance from exotic attention mechanisms. Tencent built the model to work with vLLM and SGLang inference engines out of the box.

On efficiency

The 295B total parameter count breaks down to about 21B active per forward pass, a roughly 14:1 sparsity ratio typical of modern MoE designs. The 256K context window operates on vanilla GQA, which means practitioners can run Hy3 on existing infrastructure without rewriting attention kernels. The FP8 version cuts memory overhead further, making local deployment feasible on multi-GPU rigs that would struggle with dense models at this scale.

The Apache 2.0 license permits commercial use without restriction. Tencent's HuggingFace card includes configuration files for both vLLM and SGLang, and early testers report stable inference at full context length on 8×A100 setups.