Developers fail to account for the cost of SRAM when training large models. To run Qwen3-235B with 131K context support, developers need a massive hardware investment. Storing this model in SRAM requires 45 Cerebras chips. At $3 million per chip, the hardware cost reaches $135 million. Two NVIDIA DGX B200 systems with 2.8TB of memory handle this workload for $1 million. This 135x difference in price proves that the TCO advantage vanishes when you prioritize SRAM for massive parameter counts. Cerebras prices its inference API at $0.60 per 1 million tokens for Llama 3.1 70B, but this does not negate the high upfront capital expenditure for hardware. SRAM costs more than HBM. If you attempt to match the memory capacity of traditional GPU clusters using only on-chip SRAM, your capital costs will exceed the efficiency gains. The interconnect bandwidth of 214 PB/s provides a speed advantage, but it does not solve the problem of expensive on-chip memory. The memory bandwidth of the WSE-3 is 21 PB/s, which is 2,600 times faster than NVIDIA’s flagship Blackwell B200.
The Cerebras CS-3 provides 125 PFLOPS of AI performance in sparse FP8 precision. It contains 900,000 AI-optimized cores and 44 GB of on-chip SRAM. The memory bandwidth reaches 21 PB/s. The CS-3 die measures 46,225 mm2, which is 57 times larger than a traditional GPU. It has 4 trillion transistors. The massive memory bandwidth of 21 PB/s allows the CS-3 to run the Llama 3.1 405B model at 970 tokens per second, which is double the speed of the Blackwell architecture for single-user latency. For Llama 2 70B training, the CS-3 finishes in 1.6 minutes per billion tokens. This is 7.2x faster than an H100 cluster. For GPT-3 175B, it takes 2.3 minutes per billion tokens. This speed exceeds NVIDIA’s H100 by 4.5x. The CS-3 also runs Llama 3.1 70B at 2,100 tokens per second per user. This is 8x faster than NVIDIA’s H200. A full CS-3 rack with four chips processes 4 trillion tokens per day for Llama 3.1 405B fine-tuning. This is 2x faster than eight H100s. The WSE-3 is 52 times larger in compute cores than traditional GPUs. The WSE-3 has 4 trillion transistors, while the Blackwell B200 has 208 billion transistors. The CS-3 provides 125 PFLOPS of AI performance, which is 31 times the 4 PFLOPS of the H100. Cerebras uses weight streaming to handle models that exceed on-chip memory. This process stores model weights in external memory and streams them onto the WSE to compute each layer one at a time. MemoryX provides the large-capacity memory for these weights. The SwarmX interconnect connects multiple CS-3 systems and MemoryX units to broadcast weights and reduce gradients. This approach breaks the memory wall by moving data from memory to cores at speeds beyond traditional HBM.
The CS-4 system uses three WSE-3 Turbo processors to reach 750 PFLOPS. This architecture provides 129.6 PB/s of memory bandwidth. The system also provides 160.5 PB/s of on-chip fabric bandwidth. Cerebras handles large models poorly due to latency and bandwidth issues to main memory. You should realize that high-density compute requires liquid cooling. The CS-3 draws 15kW per chip. A full rack draws 120kW. This power draw complicates deployment for smaller data centers. The CS-4 introduces disaggregated inference, which separates prompt processing from token generation. This model uses AWS Trainium3 chips for prefill before transferring the model state to the CS-4 for decoding. The CS-4 system provides 2.0 us of I/O latency and 7.2 Tbit/s of system I/O bandwidth. The CS-3 provides 5.0 us of I/O latency and 1.2 Tbit/s of system I/O bandwidth. Will this hybrid approach satisfy the requirements for scaling in production environments? The CS-4 provides up to 2x more interactivity and 10x more token capacity per watt compared to the CS-3. The Nexus rack uses 50% fewer components than previous generations. This design reduces deployment time from days to hours. The CS-4 is built using the 5nm process. Its memory bandwidth of 129.6 PB/s is significantly higher than the 21.6 PB/s seen in the CS-3. The CS-4 system delivers 750 PFLOPS, which is much higher than the 125 PFLOPS of the CS-3.
