SN40L Memory and Dataflow
SambaNova’s SN40L Reconfigurable Dataflow Unit (RDU) manages massive AI models through a three-tier memory system. This design integrates on-chip distributed SRAM, on-package HBM, and off-package DDR DRAM to overcome the memory wall. The SN40L utilizes a 5nm TSMC process and a 2.5D Chip-on-Wafer-on-Substrate (CoWoS) chiplet-based design to handle models with up to 5 trillion parameters and sequence lengths exceeding 256k. Groq’s LPU architecture requires hundreds of chips to run a single 70B model because each LPU only carries 230 MiB of memory. SambaNova’s 70B inference configuration uses only 16 chips. The SN40L RDU architecture manages large-scale AI workloads by combining tensor parallelism across chips and pipeline parallelism within each chip to deliver a compute roofline of 10.2 bf16 PFLOPS when using 16 interconnected chips in a peer-to-peer network. This setup avoids the repeated memory transfers that slow down traditional instruction-based processors. The SN40L RDU architecture yields higher operational intensity for activations. The 1040 distributed Pattern Memory Units (PMUs) provide hundreds of TBps of on-chip memory bandwidth. Groq’s LPU focuses on linear algebra compute and uses a programmable assembly line architecture, but its over-reliance on expensive SRAM limits scalability.
Scaling Agentic AI and Efficiency
SambaNova achieves 10x better inference performance at one-tenth the hardware footprint than GPU alternatives. A single SambaNova rack consumes between 8 kW and 15 kW, averaging 10 kW. Groq requires tens of racks and massive power to run each model instance. I find the complexity of the RDU architecture a steep learning curve for small teams. You know the difficulty of scaling LLMs without massive power increases. The SN40L delivers high tokens-per-second (TPS) for models like OpenAI’s GPT-OSS 120B. The SN40L also handles the Llama 400B model at 115 tokens per second using less than 19 kW of power. The newer SN50 RDU targets agentic AI workloads. SambaNova secured 350 million dollars to support these developments. The SN50 delivers performance up to five times faster than competitive chips while cutting inference costs by up to threefold. This collaboration with Intel combines Intel’s leadership in compute, networking, and memory with SambaNova’s full-stack systems. The SN50 delivers faster token generation and better system throughput for multi-step reasoning workloads. I wonder if the hardware costs will eventually limit the adoption of the SN50 in smaller data centers.
Enterprise Deployment and Cloud
DataScale combines RDU processors with the SambaFlow software stack and high-speed interconnect fabric. This platform supports terabyte-scale memory capacities to run models with hundreds of billions of parameters. SambaNova’s $11 billion post-money valuation follows a Series F round involving General Atlantic and Intel Capital. Samba-1, the flagship trillion-parameter model, uses a Composition of Experts (CoE) architecture with over 50 specialized models. This CoE design yields 10x better inference efficiency than monolithic models. Samba-1 uses an intelligent router to direct queries to the most appropriate expert models. SambaManaged allows for deployment in just 90 days. SambaCloud, available on AWS Marketplace since May 2025, delivers fast inference on models like DeepSeek R1 671B and Llama 4. SambaCloud also released native Python and JavaScript/TypeScript SDKs in October 2025. SambaStudio manages the complete AI model lifecycle, including training, fine-tuning, and deployment. This system allows users to manage AI workloads across the organization. Organizations like Argonne National Laboratory, Lawrence Livermore National Laboratory, and Los Alamos National Laboratory use SambaNova systems for advanced research.
| SN40L Metric | Value |
|---|---|
| Parameter Capacity | 5 Trillion |
| Sequence Length | 256k+ |
| Memory Tiers | SRAM, HBM, DDR DRAM |
| Average Power | 10 kW |
| 70B Configuration | 16 Chips |
Customers include Argonne National Laboratory, Lawrence Livermore National Laboratory, Los Alamos National Laboratory, Oak Ridge National Laboratory, and Texas Advanced Computing Center (TACC).




