SambaNova’s SN40L RDU generates 140 tokens per second and consumes 10KW per rack, which is similar to a CPU server. This hardware enables the execution of Llama 3.1 405B at full precision with 132 tokens per second. The platform uses "composition-of-experts", which allows native virtualization of various checkpoints on a single platform. This approach helps companies avoid the need for one rack of GPU per model checkpoint. If a company fine-tunes a model and creates 100 different checkpoints, they would need 100 racks of GPU if they used Nvidia hardware. Every incremental benefit from a GPU comes at an increasing cost in acquisition dollars, power, and complexity. Nvidia racks consume 140 kilowatts, a figure that grows, which forces data center players to upgrade electrical infrastructure and potentially requires a nuclear power plant. SambaNova positions its RDU as a power-efficient alternative meant for better inferencing speed and accuracy than Nvidia GPUs.
The composition-of-experts method addresses a fundamental scaling problem for enterprises. Most firms do not want to incur the cost of training a frontier model from the ground up. Instead, an open-source pre-trained model fine-tuned on private data achieves higher accuracy at 5% of the cost needed for training. SambaNova serves customers like Saudi Aramco, Accenture, and Analog Devices. The company holds a $5.1 billion valuation and receives backing from investors like SoftBank and Blackrock. Liang stated that the world is moving to production and that billions of training dollars will be the prerogative of a select few companies like Meta, OpenAI, and Google.
Groq LPU latency and memory constraints
Groq maintains a lead in raw Time to First Token performance through its LPU architecture. In December 2024, Groq achieved 276 tokens per second on Llama 3.3 70B. In February 2024, benchmarking firm ArtificialAnalysis clocked Groq at 241 tokens per second on Llama-2 70B, though the LPU relies on approximately 230 MB of on-chip SRAM without external HBM or DRAM. Because each LPU holds only 230 MB of on-chip SRAM and lacks external HBM or DRAM, serving a large model requires sharding across many chips, which creates a genuine cost-per-token vulnerability at high throughput. This architecture delivers up to 80 TB/s of bandwidth.
The deterministic, compiler-scheduled execution of the LPU produces each token as fast as the hardware’s memory bandwidth allows. This design minimizes memory access overhead per token. However, the tiny per-chip memory limits the model catalog to models Groq has compiled for its hardware. Serving Mixtral 8x7B required 576 GroqChips across 8 racks, 9 servers, and 8 chips per server. This massive chip requirement contrasts with the ability of a single H100 to hold the same model at low batch sizes. In December 2025, Nvidia effectively neutralized the threat from Groq when it licensed its inference IP and hired its founders, including Jonathan Ross and Sunny Madra, in a deal reported at roughly $20 billion.
| Feature | SambaNova SN40L | Groq LPU |
|---|---|---|
| Llama 3.3 70B Speed | Not specified | 276 tokens/sec |
| Llama 3.1 405B Speed | 132 tokens/sec | Not specified |
| On-chip memory | Not specified | ~230 MB SRAM |
| Power per rack | 10KW | Not specified |
Evaluation of real-time inference performance
You should analyze the specific latency budget for your application before selecting a provider. For AI voice agents, the total target latency must stay under 500ms end-to-end. This budget includes STT under 80ms, NLU under 50ms, LLM inference TTFT under 200ms, TTS under 120ms, and audio transport under 50ms. Groq’s P95 TTFT ranges from 200 to 400ms, which remains marginal for voice applications and requires prompt optimization. Fireworks AI offers a higher P95 TTFT of 350 to 700ms. If a provider has a P50 TTFT of 180ms but a P95 of 800ms, one in twenty voice AI turns will have an 800ms pause before the AI starts speaking.
SambaNova targets the enterprise with a full-stack approach including SambaOrchestrator software and SambaRack systems. Groq targets the neocloud market through GroqCloud. Groq serves open-weight models such as Llama 3.1 8B and Mixtral. SambaNova provides free API access to developers to encourage experimentation with 405B and 70B models. SambaNova builds "token factories" in India to act as fine-tuning checkpoints.
Will the memory limitations of the LPU prevent Groq from capturing the largest enterprise models? SambaNova’s SN40L capacity for large memory provides the type of privacy enterprises want for security reasons.
