Samsung HBM4 24-Hi: The First 128GB Stack is Here—And AI Training Will Never Be the Same

HBM4 was always about density and bandwidth, but Samsung’s latest 24-Hi stack is a genuine milestone: 128GB per stack, 2.5TB/s bandwidth, and—crucially—a process node that keeps thermal/power in check. For AI hardware engineers and ML infra folks, this is the moment when memory bottlenecks start to genuinely fade for multi-trillion parameter training runs.

Why 24-Hi HBM4 Changes the Game

AI accelerators (think NVIDIA’s B100 or AMD’s MI400 series) are memory-bound: memory bandwidth and capacity are the bottlenecks for large sequence lengths and big batch training. With 128GB per stack, you can now fit >1TB of HBM on a single accelerator board (using 8 stacks). This means training longer-context LLMs, bigger vision models, and more complex agentic systems without sharding or heavy off-chip swap.

There’s more: the new process uses advanced TCB (thermal compression bonding), keeping the 24-layer stack cool enough to avoid the bandwidth throttling that plagued early 16-Hi and 12-Hi chips. For system builders and datacenter architects, it means denser racks, lower latency, and less exotic liquid cooling—at least for now.

What Engineers Need to Watch

Engineers should pay attention to firmware and controller updates—HBM4 24-Hi introduces new timings and error correction paths. Also, expect software changes: ML frameworks (like PyTorch and JAX) are already adding primitives to leverage lower latency and greater in-memory batch sizes. I’d also watch for cost ripple effects: more memory per board could make DIY training clusters more viable for startups, not just hyperscalers.