Cerebras CS-3 wafer-scale engine and sparse MoE training explained

Cerebras Systems builds the WSE-3 using nearly an entire 300mm silicon wafer to create a single 46,255 mm2 chip. This monolithic design integrates 4 trillion transistors and 900,000 AI-optimized cores directly onto one piece of silicon. I find the scale of this hardware impressive because it bypasses the communication bottlenecks that hit traditional GPU clusters. Most manufacturers cut wafers into small dies, but Cerebras keeps the wafer intact and uses architectural innovations to route around defects. The WSE-3 uses 5nm technology from TSMC and delivers 125 petaflops of peak AI performance.

The CS-3 system pairs this wafer with MemoryX external memory, which reaches up to 1.2 PB for hyperscale configurations. This memory capacity allows the system to store neural network models with up to 24 trillion parameters in a single logical memory space. You should understand that this eliminates the need for the software partitioning or sharding tricks that other computers require to handle massive models. Because the memory resides on the same silicon as the compute cores, the WSE-3 provides 21 PB/s of memory bandwidth. This figure exceeds the 3 TB/s HBM bandwidth of an NVIDIA H100 by 7,000x. The system consumes 23kW of peak power, which exceeds the 14.3kW draw of an NVIDIA DGX B200.

Monolithic silicon and sparsity

The WSE-3 architecture uses Sparse Linear Algebra Cores to manage data more efficiently than dense execution engines. These cores skip multiplications by zero, which saves power and time since neural networks often contain many zero values. Cerebras calls this sparsity harvesting technology. The design also uses a 2D mesh network to connect the 900,000 cores, providing 214 Pb/s of aggregate fabric bandwidth. This on-wafer wiring reduces the latency seen when moving data between separate chips in a cluster.

The system handles heterogeneous data shapes through the Variable Tensor Shape feature in the Cerebras Software Platform. This feature strips away padding from shorter sequences in a dataset, which prevents the processor from wasting FLOPs on empty tokens. In a test using the Transformer-large model on the WMT 2014 English to German dataset, this method produced a 5x speedup. The architecture also supports Mixture of Experts (MoE) and vision transformers. The WSE-3 delivers 125 petaflops of AI compute using industry standard FP16 precision.

Scaling for massive models

Cerebras scales its capacity by combining up to 2,048 CS-3 systems via the SwarmX interconnect. A cluster of these systems can achieve 256 exaFLOPS of AI compute. This scaling approach avoids the communication overhead that usually degrades performance in large GPU clusters. The CS-3 can train a Llama 70B model in one day when using a configured cluster. The system supports training models up to 24 trillion parameters without needing to partition the model across different devices.

The disaggregated memory architecture uses MemoryX to store weights, which uses flash and DRAM to pipeline load/store requests. This allows the CS-3 to load a 1 trillion parameter model using a single 1.2 PB rack. The CS-3 provides 2.5x the inference speed of the NVIDIA DGX B200 when running the 400-billion parameter Llama 4 Maverick model. While the hardware delivers high throughput, the software ecosystem remains smaller than the NVIDIA CUDA environment. The WSE-3 provides 44 GB of on-chip SRAM, which is 880x more than the 0.05 GB found on an NVIDIA H100. Can Cerebras maintain this performance lead as NVIDIA releases more Blackwell iterations? The CS-3 delivers 2.2x better performance per watt than the DGX B200 despite drawing more total power.