Model capabilities and training
Meta released the Llama 3.2 90B Vision model on September 25, 2024. This multimodal model architecture builds on the Llama 3.1 text-only model. It uses a separately trained vision adapter that feeds image encoder representations into the core LLM through cross-attention layers. The model performs tasks like visual recognition, image reasoning, and image captioning. Meta pretrained the model on 6B image and text pairs and used over 3M synthetically generated examples for instruction tuning. The instruction-tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety. The pretraining data has a cutoff of December 2023. For image and text applications, the model supports only English. The model family uses Grouped-Query Attention (GQA) to improve inference scalability. Users can perform visual grounding, which allows the model to pinpoint objects or regions in an image based on natural language descriptions. The model also enables image-text retrieval and document visual question answering. For text-only tasks, the model supports English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai. The training process used 2.02M GPU hours of computation on H100-80GB hardware. This training generated 584 tons of CO2eq.
Solving the data center power crisis
SambaNova’s SN40L Reconfigurable Dataflow Unit (RDU) provides the infrastructure to run these models in enterprise data centers. This fourth-generation chip uses TSMC 5nm technology and delivers 688 FP16 Tflop/s. The RDU architecture employs a three-tier memory system: 520 MiB of on-chip distributed SRAM, 64 GiB of co-packaged HBM, and up to 1.5 TiB of DDR DRAM. This configuration helps bypass the memory wall as it provides high-capacity memory for storing parameters of many expert models. I find the low power footprint of the SN40L particularly useful for avoiding the massive costs of new data center builds. Existing enterprise data centers typically support 5 kW to 8 kW for traditional workloads or up to 30 kW for air-cooled racks. Next-generation GPU systems like the NVIDIA Blackwell DGX GB200 NVL72 draw 120 kW per rack. The Rubin Ultra NVL576 consumes as much as 600 kW per rack. Because the SN40L runs at an average of 10 kW per rack, it fits into existing facilities that support only 30 kW or less, bypassing the need for expensive liquid cooling or new construction. One report estimates that power shortages will delay or cancel 30% to 50% of AI data centers planned for 2026. The construction of a 1 GW AI data center to power NVIDIA VR200 NVL72 systems costs approximately $47 billion and takes about three years to complete.
Performance and deployment economics
SambaNova’s hardware delivers advantages for Composition of Experts (CoE) workflows. These workflows use many small, specialized models instead of monolithic LLMs. In an 8-socket RDU Node configuration, the SN40L reduces machine footprint by up to 19x. It also improves model switching times by 15x to 31x. The SN40L achieves a 3.7x speedup over the DGX H100 and a 6.6x speedup over the DGX A100 for these inference deployments. I would skip the massive GPU clusters if you need to deploy agentic AI without a multi-year construction project. The SN40L delivers 3X the speed and 5X the efficiency of leading GPUs. You can port your applications to SambaNova in minutes because the APIs are OpenAI compatible. The SN40L uses a reconfigurable dataflow processor that maps the AI model graph to the chip to optimize inference. This architecture uses on-chip distributed SRAM, on-package HBM, and off-package DDR DRAM to handle models with over 10 trillion parameters and context lengths exceeding 10 million tokens. I find the 3.7x speedup over the DGX H100 for CoE workloads to be a decisive metric for enterprise deployment. Will the SN40L maintain its efficiency advantage as newer Blackwell models enter widespread production?
| Feature | SN40L Specification |
|---|---|
| Chip Process | TSMC 5nm |
| Peak Compute | 688 FP16 Tflop/s |
| On-chip SRAM | 520 MiB |
| Co-packaged HBM | 64 GiB |
| DDR DRAM | Up to 1.5 TiB |
| Rack Power | 10 kW average |
SambaNova recently secured $350 million in Series E funding to expand its production of SN50 hardware and cloud services. The company valuation reached $5.1 billion following its Series D round. SoftBank has already selected SambaNova DataScale to power its next-generation AI data centers in Japan.
