DeepMind’s Rhea-2: Sparse Mixture-of-Experts at 400B Parameters—But Actually Efficient

DeepMind has revealed Rhea-2, their newest sparse Mixture-of-Experts (MoE) LLM. On paper: 400B parameters, SOTA on multitask and reasoning benchmarks, and—most interestingly—training and inference costs under what you’d expect for a 70B dense model. How? This is the first MoE that uses adaptive routing and periodic expert pruning, letting it scale up efficiently without ballooning hardware cost.

What’s New Technically?

Most MoE models suffer from ‘expert bloat’—tons of parameters that rarely activate, wasting VRAM and compute. Rhea-2 uses a dynamic router (a tiny LSTM conditioned on activations) to pick experts per token, plus a scheduled “dead head” pruning pass every 10K steps. That means the model stays lean, but still learns the rare, weird stuff that makes LLMs useful in the real world.

Why Should Engineers Care?

This isn’t just about leaderboard chasing. Efficient scaling means you can train bigger models on smaller clusters—or squeeze massive models into edge deployments you’d never consider before. Developers will see faster inference, lower costs, and—crucially—much more predictable behavior as you scale batch sizes up or down.

The code for the router and pruning scheduler is already out under Apache 2.0, with PyTorch and JAX reference versions. Expect to see a wave of open-source MoE models using Rhea-2’s tricks in the next few months.

The Takeaway

We’re not just stacking more parameters for hype anymore. This is the first real step toward sustainable, usable trillion-parameter LLMs that regular shops can actually run.