SN40L performance and trillion-parameter expert scaling

RDU scaling for large models

The SN40L RDU node running 150 7B experts totaling over 1 trillion parameters achieves a 3.7x speedup over a DGX H100. I find the 19x footprint reduction and the 31x model switching time more impressive than the raw compute numbers. An 8-socket RDU node delivers 638 BF16 TFLOPS of peak compute performance using 1040 distributed Pattern Compute Units. This setup handles the massive data flows of trillion-parameter models more effectively than traditional GPUs. As the number of experts increases beyond 50, performance improves because memory limitations make running such models on DGX H100 or DGX A100 systems extremely slow or impossible. I see a 6.6x speedup over the DGX A100 when running these large-scale compositions. The SN40L RDU, which uses TSMC 5-nanometer technology, contains 102 billion transistors spread across 1,040 cores. This 2.5D Chip-on-Wafer-on-Substrate design includes two SN40L Reconfigurable Dataflow Dies. The hardware uses a coarse-grained reconfigurable array of dataflow cores connected by a two-dimensional mesh interconnect called the Reconfigurable Dataflow Network. The hardware includes 1040 distributed Pattern Memory Units that provide hundreds of TBps of on-chip memory bandwidth along with high bank-level parallelism. SambaNova’s training performance for GPT reached 2.1x faster throughput than Nvidia using Dataflow-as-a-Service.

Three-tier memory and dataflow

The SN40L RDU uses a three-tier memory architecture to place data according to access frequency. It utilizes 520 MiB of on-chip distributed SRAM, 64 GiB of on-package HBM, and 1.5 TiB of off-package DDR. Loading models from DDR to HBM reaches 1 TB/s in a single SN40L node. Traditional accelerators send data back to memory between operations, which adds latency and wastes movement. The SN40L RDU uses streaming dataflow parallelism to aggressively fuse hundreds of complex operators and pipeline parallelism across compute units to speed up auto-regressive decoding without repeating redundant memory transfers. If you manage a data center with tight power constraints, you might notice the 10 kW draw of a SambaNova rack compared to the 140 kW draw of an Nvidia system. I note that the 64 GB HBM capacity on the SN40L lags behind the 192 GB found in the AMD MI300X. The DDR tier reaches a peak memory capacity of 1.5 TiB with a peak bandwidth of over 200 GB/s, while the HBM tier provides 64 GiB of capacity with a peak bandwidth of about 2 TB/s per socket. The device interfaces with a host x86 CPU using a PCIe link, which supports DMA between host and device off-chip memory. The SN40L RDU is well suited for existing inference deployments and power-constrained environments.

Composition of experts and sparsity

Composition of Experts (CoE) combines specialized, smaller models to match the capabilities of monolithic LLMs. The SN40L handles this by using a software compiler to divide the load between three memory layers. This design accommodates both dense and sparse compute. SambaNova uses a software layer for scheduling and data movement to exploit sparsity. This approach enables the system to run 150 7B experts in tandem. The hardware uses a tiled array of memory and compute units connected by a high-speed switching fabric. The SN40L supports sequence lengths as high as 256,000 tokens. The SN40L also supports 713 tokens per second on gpt-oss-120b for a 10k input and 1k output request. The Samba-CoE system uses 150 7B expert models, primarily derived from Llama2-7B, to work in concert. The router model dynamically assigns each input prompt to the most relevant expert, such as routing a math query to a math expert. I note that the 64 GB HBM capacity on the SN40L lags behind the 192 GB found in the AMD MI300X. Does the proprietary interconnect scale as easily to thousands of nodes as Nvidia’s InfiniBand?