Tenstorrent scaling strategies
Tenstorrent ships RISC-V based accelerators that scale from a single card to a 32-chip system. The $11,999 liquid-cooled Blackhole QuietBox, which weighs 80 pounds (36 kg), provides a development platform for the upcoming Blackhole Galaxy servers. These Galaxy servers use the same chips, memory, and interconnects to ensure seamless scaling. Jim Keller, the Tenstorrent CEO, claims that 16 Galaxy servers containing 512 chips can run DeepSeek-671B at 350 tokens per second per user at batch 32. This setup provides 56 Ethernet ports per box, whereas GPU servers typically provide only eight external ports. Tenstorrent is building 1,000 Galaxy servers and recently received an order for a 96-Galaxy cluster containing 3,072 Blackhole chips for a customer outside the U.S. The Blackhole P100a costs $999, while the P150a costs $1399, and these PCIe accelerator cards use a Blackhole SoC made of four RISC-V CPU tiles with four SiFive X280 cores per tile. A virtual UART in the OpenSBI firmware allows a console program on the PCIe host to communicate with Linux running on the Blackhole card via shared memory. Drew Fustini sent a pull request to add initial Tenstorrent vendor support for the Blackhole hardware in the Linux 6.19 kernel. Keller also notes that agentic computing is limited by scalar processing, an application of Amdahl’s Law. Tenstorrent’s largest customer remains AI& in Japan, where the CEO is former executive David Bennett. However, an immature software stack makes harnessing Tenstorrent compute challenging.
AMD deployment and software instability
AMD secures large cloud contracts for its Instinct MI300X hardware through providers like Vultr and Oracle. Oracle Cloud Infrastructure offers a bare-metal instance named BM.GPU.MI300X.8 with 8 AMD Instinct MI300X accelerators. This configuration delivers 1.5TB of HBM3 GPU memory and 5.3TB/s bandwidth, alongside 2TB of system memory and 8 x 3.84TB NVMe storage. The MI300X supports massive AI models containing billions of parameters and scales up to 16,384 GPUs in a single cluster. Oracle reports that MI300X achieved a 65 millisecond "time to first token" latency for the Llama 2 70B model and generated 3,643 tokens across 256 concurrent user requests. In another test using 2,048 input and 128 output tokens, the system delivered an end-to-end latency of 1.6 seconds. Even with an optimized stack, AMD Instinct MI300X shows 30% higher performance than Nvidia H100. If you want to avoid the high costs of Nvidia, you might look at these two paths.
| Feature | Tenstorrent P150a | AMD MI300X (per instance) |
|---|---|---|
| Price | $1399 USD | $6 per GPU/hour |
| Memory | RISC-V/DRAM | 1.5TB HBM3 |
| Bandwidth | RISC-V/DRAM | 5.3TB/s |
AMD faces significant software-driven performance volatility. A user reported a ~38% throughput regression for MiniMax-M2.5 (FP8) on the MI325X when upgrading from vLLM ROCm v0.18.0 to v0.21.0. Specifically, peak throughput fell from 2,600 tokens per second per GPU to 1,600 tokens per second per GPU between March and May 2026. This monotone decline suggests commits between those versions degraded MoE performance on the gfx942 architecture. DigitalOcean and various Neocloud operators also offer MI325X and MI300X GPUs for intensive workloads.
Comparing sparse inference efficiency
AMD hardware handles massive models like the 1.4 TiB Kimi-K3 MXFP4 mixture-of-experts model through disaggregated deployment across MI300X and MI325X GPUs. The MI355X also provides 1.9-3.3x faster decode using MXFP4. Tenstorrent approaches sparse LLM inference by handling the KV cache in the DRAM on the same chips as the decode. Jim Keller notes that Tenstorrent can connect arbitrary numbers of tensor processors together to fit tensors entirely into SRAM or stream data from DRAM. This provides a direct advantage over architectures that lack DRAM and require more hardware to scale. Tenstorrent’s design also addresses Rents Rule, which states that I/O requirements grow sub-linearly with logic, by providing 56 Ethernet ports per server. Tenstorrent’s ability to split large tensors across hundreds of chips facilitates high token rates, such as the 350 tokens per second per user achieved with the DeepSeek-671B model. This works because the architecture manages the balance of compute, memory, and I/O. Will AMD’s ROCm stack overcome the performance regressions seen in recent vLLM updates? I believe Tenstorrent comes ahead for sparse LLM inference in cloud data centers this quarter because its architecture addresses the memory and networking bottlenecks of large models more effectively than the currently volatile AMD stack.
