Baidu releases ERNIE 5.5 Flash base weights
Baidu has released ERNIE-5.5-Flash-Base, a 280B-parameter sparse MoE checkpoint from the end of pre-training, offering developers a starting point for further training without paying for pre-training from scratch.
Baidu announced ERNIE-5.5-Flash-Base, a Mixture-of-Experts language model with 280B total parameters and 14B activated per token. It was pre-trained from scratch on 33.5T tokens with a training context of up to 128K, and supports 1M-token contexts for fine-tuning and inference.
The release is the checkpoint as it stood at the end of pre-training, before any mid- or post-training. Baidu pitches it as a shared starting point for domain adaptation, instruction tuning and reinforcement-learning research, and as a clean baseline for comparing training methods under controlled conditions. The company says the pre-training corpus favors capabilities that are hard to add later, including repository-scale code and long-context understanding.
The model has 43 Transformer layers, a hidden size of 4096 and 64 attention heads. Efficiency drives the attention design. One layer in seven uses Multi-head Latent Attention (MLA) with token-level access to the full context. The remaining layers use Heavily Compressed Attention (HCA), which compresses every 128 tokens into a single cached entry and adds a short window of recent tokens. The MLA layers use sparse attention: a lightweight indexer picks the top-k positions for each query, so the cost of core attention per query stays flat as context grows. Baidu says this matches dense attention in pre-training loss and trains substantially faster.
Other changes include attention gates computed from the low-rank query latent, which uses 75% fewer gate weights, and a low-rank head-mixing step called Virtual Head Attention.
The excerpt does not include benchmark results, license terms or download details.