The SambaNova SN40L Reconfigurable Dataflow Unit (RDU) uses a three-tier memory hierarchy to execute large language models. This architecture consists of 520 MB of on-chip SRAM, 64 GB of HBM3, and up to 1.5 TB of DDR5 per socket. The RDU executes model graphs using a dataflow execution model. SambaNova Composer compiles the model graph into a static graph that pipelines a single request through the hardware. The RDU reads weights from the specific memory tier that holds them during execution. This structure allows the system to handle models that exceed the capacity of standard HBM.
The memory architecture and MoE advantage
The three-tier memory hierarchy provides a specific advantage for Large Mixture of Experts (MoE) models. A model like Llama 4 Maverick has 128 experts and 400 billion total parameters, but only 17 billion parameters are active during a forward pass. On a standard GPU cluster, you must spread the full model across enough HBM to fit every expert simultaneously. The SN40L keeps cold experts in the DDR5 tier and fetches them on demand. This design eliminates the need for an 8x NVLink tensor-parallel configuration to achieve memory fit.
I see the SN40L excel with these MoE workloads at low concurrency. A single 8-RDU node serves Llama 4 Maverick at 2,800 tokens per second at batch 1. In this scenario, the DDR5 expert spill allows the node to handle the model without tensor parallelism complexity. This capability also applies to long-context decode workloads. Because the DDR5 tier provides capacity for models between 141 GB and 512 GB, teams avoid the configuration complexity of a four-way H200 tensor parallel setup when running long-context decode workloads at very large scale.
Samba-CoE, a system with 150 experts and a trillion total parameters, demonstrates this scaling. The system stores experts in DDR, keeps the router in HBM, and moves the selected expert into HBM for execution. This modular approach reduces the machine footprint by up to 19x for CoE inference deployments. It also speeds up model switching time by 15x to 31x. An 8-socket RDU node achieves a 3.7x speedup over a DGX H100 and a 6.6x speedup over a DGX A100.
Constraints in dense model performance
The advantages of the DDR5 tier disappear when I run dense models. Dense transformers activate every parameter on every token pass. In these cases, HBM bandwidth determines the performance limit. The H200 provides 4.8 TB/s of bandwidth and the B200 provides 8.0 TB/s of bandwidth. The SN40L node has an estimated effective HBM bandwidth of approximately 2 TB/s.
The difference in throughput is clear during high-batch dense workloads. At batch 32 on the Llama 3.3 70B model, the B200 delivers approximately 6,100 tokens per second. The SN40L node delivers approximately 2,100 tokens per second. A single H200 also outperforms or matches the 8-RDU SN40L node on Llama 3.3 70B at batch 8 and beyond. Dense models do not benefit from the cold storage capacity of DDR5 because every parameter remains active.
| Specification | SambaNova SN40L (1 RDU) | NVIDIA H200 (SXM5) | NVIDIA B200 (SXM6) |
|---|---|---|---|
| Memory Tiers | SRAM, HBM3, DDR5 | HBM3 | HBM |
| On-chip SRAM | 520 MB | N/A | N/A |
| HBM Capacity | 64 GB | 141 GB | N/A |
| DDR5 Capacity | 1.5 TB per socket | N/A | N/A |
| HBM Bandwidth | ~2 TB/s | 4.8 TB/s | 8.0 TB/s |
| Llama 4 Maverick (Batch 1) | 2,800 tokens/sec | N/A | N/A |
| Llama 3.3 70B (Batch 32) | ~2,100 tokens/sec | N/A | ~6,100 tokens/sec |
Latency and concurrency realities
The dataflow model compiles models into a static graph executed from a pre-warmed state. There is no per-request weight loading from HBM into compute cores. This provides low Time to First Token (TTFT) on large MoE models at batch 1. For Llama 4 Maverick at batch 1, SambaNova publishes a TTFT under 100ms. An H200 x8 vLLM setup at batch 1 on the same model typically shows a TTFT between 300ms and 700ms.
However, the performance trend reverses as concurrency increases. At batch 32, the inter-token latency (ITL) flips in favor of NVIDIA. The higher HBM bandwidth of the H200 and B200 sustains lower per-token latency under load. The SN40L ITL degrades more at high concurrency because the DDR5 fetch bandwidth is fixed and shared.
I also see different results when testing DeepSeek R1. The SN40L system reaches a peak output speed of 269 tokens per second for single-user workloads. This is the highest speed measured for DeepSeek R1, and it is more than 3x the peak output speed of the NVIDIA B200 using TensorRT-LLM. The SN40L system handles batch sizes of up to 256, reaching a system throughput of approximately 4,700 tokens per second at and beyond 256 concurrent requests.
The cost per million token calculation
The cost-per-million-token (CPM) calculation determines the financial viability of these platforms. For a GPU cloud, the formula is: CPM = ($/hr per node) / (tokens_per_sec x 3600 / 1,000,000). SambaNova Cloud uses a direct per-token price from their API. There is no hourly rate, no batch-scaling discount, and no spot pricing. The managed API charges per output token regardless of concurrency.
At batch 32, the GPU cloud on Spheron delivers Llama 4 Maverick at $0.47/M tokens on spot and $1.39 to $1.52/M tokens on-demand. SambaNova Cloud charges approximately $6/M output tokens for the same model. The GPU cloud advantage at scale is substantial. You should ignore the SambaNova argument if you already run production GPU infrastructure. The SambaNova value proposition focuses on zero infrastructure overhead, no Docker, no vLLM configuration, and no CUDA driver management.
Software limitations and specialized hardware
The RDU architecture requires specific software to function. SambaNova Composer is the proprietary layer that compiles the model graph. Unlike CUDA, which accepts arbitrary kernel code, Composer only supports architectures it can map to the RDU dataflow. This means there is no support for vLLM, SGLLang, or custom attention kernels. Users access the hardware through the SambaNova Cloud API instead of a bare-metal SSH session.
The RDU is an inference-only device. The architecture has no gradient computation path in the released tooling. This makes it unsuitable for teams that need to perform training and inference on the same hardware fleet. Additionally, SambaNova Composer does not support vision encoders as of May 2026. Can SambaNova extend its proprietary software support to include vision encoders and advanced speculative decoding frameworks as quickly as the rapid updates to the NVIDIA CUDA ecosystem arrive for modern, high-performance GPU hardware?
Deployment and energy density
Deployment density is a significant factor for enterprise data centers. The SN40L rack consumes an average of 10 kilowatts. This is much lower than the 140 kilowatts required by an NVIDIA rack. The SN40L uses air cooling, which allows it to fit into existing data center facilities. This differs from the NVIDIA Rubin GPUs, which require liquid cooling.
The RDU also allows for high model density. The DDR5 tier provides enough capacity to host hundreds of heterogeneous models or checkpoints on a single socket. This allows for quick model swapping without being bottlenecked by host PCIe bandwidth. The 16 chips of an SN40L configuration use a peer-to-peer network to achieve 10.2 BF16 PFLOPS of performance.
Final verdict for enterprise deployment
The decision to use the SN40L depends on the specific workload requirements. I recommend the SN40L for organizations that need low-latency inference for large MoE models at low concurrency. It is also a strong choice for long-context decode tasks where the DDR5 tier can handle KV cache overflow. The zero-ops managed API is a valid choice for teams that lack dedicated ML infrastructure engineers and want to avoid managing CUDA drivers or Docker containers.
I recommend NVIDIA H200 and B200 for teams that need high-batch throughput for dense models. The bandwidth advantages of the H200 and B200 are too large to ignore when serving many concurrent users. NVIDIA remains the only choice for teams that require training, fine-tuning, or the ability to run custom kernels from frameworks like vLLM or SGLang.
For enterprises running large, specialized agentic workflows that require many small models, the SN40L provides a compelling density and power profile. For standard, high-volume dense transformer workloads, the GPU cloud remains the more cost-effective and performant option.
