Why the SambaNova SN40L RDU beats the NVIDIA H200 on inference economics

The memory wall and expert models

I see buyers losing money when they treat inference like training. AI model training is a data processing challenge, whereas AI inference is a data flow challenge. NVIDIA H200 GPUs fetch weights from memory for every token. In this decode phase, the trip to memory takes much longer than the computation. This cycle means the GPU spends most of its time in a wait state for memory. SambaNova’s SN40L RDU uses streaming dataflow parallelism to move model operations across the chip. Because the SN40L uses streaming dataflow parallelism and a three-tier memory system, it delivers 9x greater performance and 5.6x more energy efficiency than the NVIDIA H200 at low batch sizes. This architecture allows the SN40L to use 85% of its memory bandwidth for weights. Each chip can address the memory of all 16 chips in a rack, which lets a single rack serve models with several hundred billion parameters. Composition of Experts deployment challenges include low Operational Intensity and the need for high memory capacity to keep many experts in memory. Small, expert-tuned models cause low GPU utilization. Small, expert-tuned models make AI more accessible, but current GPUs struggle with the frequent model switching required for these workflows as they cannot handle the intensity of many expert changes. I would avoid the H200 if your workload relies on switching between many small expert models frequently.

Infrastructure and power constraints

Data centers face a power crisis because next-generation GPUs require massive energy. NVIDIA Blackwell NVL72 needs up to 120kW per rack and requires exotic liquid cooling. The upcoming Vera Rubin family needs up to 600kW per rack. Most existing data centers cannot host these systems without multimillion-dollar overhauls. SambaNova’s SN40L is air-cooled and fits into existing data center infrastructure. A typical SambaNova rack draws between 7 and 14.5 kW in inference mode. During tests of the Meta Llama 3.3 70B model at low batch sizes, the SN40L delivered 9x greater performance and 5.6x more energy efficiency than the NVIDIA H200. Even at higher batch sizes, the SN40L is 4x faster and 2.5x more energy efficient than the H200. Do you really want to spend millions on liquid cooling overhauls just to run inference?

The industry is shifting from training to inference. SambaNova raised $350M for its SN50 inference chip and cloud platform. SoftBank will be the first customer to deploy the SN50 in its next-generation AI data centers in Japan. The SN50 links up to 256 accelerators over a multi-terabyte-per-second interconnect. This capability supports larger batch sizes and helps users deploy bigger, longer-context AI models.

Quantization and throughput errors

Efficiency comes from reducing hardware requirements before you look at throughput. Hivenet’s NVFP4 configuration for Qwen3.6-27B uses half the hardware needed for full precision. This configuration uses mixed precision to keep sensitive parts of the model at higher precision. The lowest reported capability retention for this setup was 95.5% on LiveCodeBench. For the Llama 3.3 70B model, the SN50 RDU provides 895 tokens per second per user at FP8 with 1K input and 1K output. This compares to 184 tokens per second per user on the NVIDIA B200. If you ignore the impact of quantization on model quality, you might buy hardware that fails your specific task.

I also note that NVIDIA reframes its value around total cost of ownership. Their TensorRT-LLM software claims a 3.55x throughput speedup for Llama 3.3 70B on H200 hardware. This gain comes from speculative decoding, in-flight batching, KV caching, and custom FP8 quantization. However, specialized chips like the SN50 provide 3X lower total cost of ownership for agentic AI. The SN50 delivers five times more compute per accelerator and four times more network bandwidth than the previous generation. In terms of costs, Nemotron 3 Super 120B costs $1,000 per million tokens for 1M requests per month using 2,000 input tokens and 500 output tokens per request. This is cheaper than the $1,150 per million tokens for GPT-OSS 120B. For comparison, Llama 4 Maverick costs $1,750 per million tokens. AWS Inferentia 2 costs approximately $12.50 per million tokens. Will NVIDIA’s software optimizations keep pace with these specialized hardware gains?