Deploying open-source LLMs with Tenstorrent Wormhole and TT-Metalium

Wormhole n150 hardware specifications

The Wormhole n150 uses the Tenstorrent Tensix Processor and operates at up to 160W. This processor includes a compute unit, network-on-chip, local cache, and "baby RISC-V" cores. Each Tensix core contains a RISC-V data movement processor, an 8×8 BF16 matrix engine, and a vector unit with 1.5 MB of local L1 SRAM. The n300 card contains 128 Tensix cores distributed as 64 cores per ASIC, which provides 192 MB of aggregate on-chip SRAM and uses GDDR6 to support much larger model weights during intensive, complex, and extremely demanding, large-scale, real-time AI inference tasks. While NVIDIA H100 SXM5 provides 80 GB HMB3 with 3.35 TB/s bandwidth, a single n300 card provides 24 GB GDDR6 with 576 GB/s bandwidth. The n300 card has 128 Tensix cores and 24 GB of GDDR6 memory. The Blackhole p150 developer card provides 32 GB of GDDR6 and operates at up to 300W, while the Blackhole p100a has 120 Tensix cores and 28 GB of GDDR6. Every Wormhole chip includes 16 Ethernet cores running at 100 Gbps each for multi-chip interconnect. You should check if your specific workload requires the higher memory capacity of the Blackhole p150, which provides 32 GB of GDDR6.

Software stack and model performance

The TT-Metalium SDK provides the low-level kernel programming model, while the TT-NN library provides Python bindings for ML engineers. Developers use these tools to port models from HuggingFace to Tenstorrent hardware. For instance, the MuseGlimmer-30B bring-up on a single Blackhole p150 achieved 119.99 AR tok/s at short context and 1,549 prompt tok/s during a 128,000-token prefill. This deployment used BFP8 for attention and BFP4 for MLPs, and the custom packed_kv_update kernel was upstreamed into TTNN. The Kimi-Linear-48B-A3B model on four n300 cards reached 13.5 tok/s at 16K context once it used paged flash-MLA on the compressed latent. This model builds fused programs from Python through ttnn.generic_op and a ProgramDescriptor for its KDA, MoE, and MLA layers. Before this change, decode went from 1.44 tok/s eager to 13.5 tok/s. The TT-Granite port of IBM’s Granite-4.0-H models on four chips achieved 10.65 – 10.69 tok/s for the tiny version. This performance is approximately 18% above an A100. The small version on eight chips reached only 5.84 – 5.89 tok/s, which is about 20% below an A100. The tt-awesome repository provides a curated collection of tools and models for the ecosystem. The tt-finetune package supports LoRA, rsLoRA, LoRA+, DoRA, and IA3 on a single Blackhole p150a device. This package includes a static planner that reports memory admission before compilation, deterministic checkpoint/resume, and standard PEFT adapter export. The tt-model-bringup project covers Qwen3.6-27B, Qwen3.6-35B-A3B MoE, Gemma 4 12B, and Nemotron-3 Nano 30B-A3B.

Optimizing for prefill and decode

Inference performance depends on how the model handles prefill and decode stages. Prefill processing of input tokens depends heavily on compute performance, but generating one token at a time during decode depends on memory bandwidth. For a dense model, a rough estimate for decode speed is memory bandwidth in GB/s divided by the weight data read per token in GB. A Qwen3 8B configuration with 36 layers, eight KV heads, and 128 head dimension uses 147,456 bytes per token with FP16 elements. For a conventional transformer, the cache for one sequence is 2 x layers x KV heads x head dimension x bytes per element x cached tokens. The DeepSeek-V4.1-Flash model on four n300 cards reached 1.68 tok/s for decode and 95 tok/s for prefill. This system runs a 552B model plus a 196B Engram and uses the host as a 449 GB expert store of pre-packed .tensorbin tiles. This deployment includes 128K context, persistent prefix state, and vision input. The Qwen3.8-27B model on two P150A cards linked by QSFP-DD reached 118 committed tok/s at 4K context and 50 tok/s at 64K context. One project reported that the 200 tok/s target for Qwen3.8-27B was not reached. Can the software stack close the gap in device-side dispatch? The Wormhole n150 is a viable option for developers seeking an auditable stack for local LLM inference.