AMD MI350X challenges Blackwell in training clusters

CDNA 4 and the MI350 series hardware

AMD’s Instinct MI350 series uses the 4nm CDNA 4 architecture to increase compute performance. The MI355X delivers 288 GB of HBM3E memory alongside 8 TB/s of bandwidth. This hardware includes 1024 matrix cores and native support for FP4 and FP6 data types. The Local Data Share capacity is 160 KB, which doubles the read bandwidth to 256 bytes per clock. These improvements provide a 4x lift in inference performance compared to the MI300X. The MI350X uses air cooling, while the MI355X uses liquid cooling to handle peak performance. The architecture consists of eight accelerator compute dies, known as XCDs, which sit atop four base dies. These base dies implement 256 MB of memory side cache and utilize Infinity Fabric for coherent memory access. Unlike the MI300X which used 5nm and 6nm processes, the MI350 series moves to a 4nm node. The CDNA 4 architecture rebalances execution units to target matrix multiplication with lower precision data types. This rebalancing aims to match NVIDIA’s Blackwell SMs in FP6 performance. The MI300X featured 19,456 cores and 192 GB of HBM3 memory with an 8192-bit memory bus. The CDNA 4 architecture also improves the L2 cache by allowing it to writeback dirty data and retain a copy of the line. The increased Local Data Share capacity of 160 KB allows kernels to allocate more space without decreasing occupancy. This improves the efficiency of many machine learning workloads. AMD also reported raising the price of its high-end AI chips to match competition. The MI350 series is designed to drop into existing AMD platforms built on the UBB infrastructure. You should evaluate the HBM3E capacity before selecting your cluster components.

Comparing MI355X to NVIDIA Blackwell

The MI355X outperforms the NVIDIA B200 in specific Large Language Model tasks. The MI355X delivers 1.3X higher inference throughput than the B200 during FP4 inference tests on Llama 3.1 405B and DeepSeek R1 when using the vLLM engine for these specific workloads. When running Llama 2-70B-LoRA in FP8, the MI355X achieves 1.13X faster time-to-train than the B200 and 1.12X faster time-to-train than the NVIDIA GB200. AMD claims the MI355X provides 40% better "tokens-per-dollar" than the B200 for Llama 3.1 405B inference in FP4. NVIDIA’s Blackwell B200 is built on a 4nm TSMC process and carries 180 GB of capacity with 7.7 TB/s of bandwidth. NVIDIA SMs provide twice as much per-clock throughput as a CDNA 4 CU across 16-bit and 8-bit data types. The MI355X maintains a lead in vector performance because of higher core counts and higher clock speeds. The MI325X provides 1.3X better compute performance than the H200 and 2x the memory capacity. Can AMD maintain this lead as NVIDIA updates its software?

The deployment ecosystem and ROCm

Major organizations like Meta, Microsoft, Oracle Cloud Infrastructure, and OpenAI deploy the Instinct MI350 series. Meta signed a multi-year agreement to deploy up to 6 gigawatts of custom AMD Instinct MI450 GPUs, a deal estimated between $60 billion and $100 billion. ROCm 7 provides a 3.5X uplift in inference performance and a 3X uplift in training performance over ROCm 6. The software supports PyTorch 2.7.0 and includes native support for distributed engines like vLLM and SGLang. ROCm 7.2.4 also supports the Radeon RX 9000 series. AMD also provides the Helios AI server platform to serve as an alternative to NVIDIA’s infrastructure. The upcoming Helios rack will feature the MI400 series GPUs with 432 GB of HBM4 memory and 19.6 TB/s of bandwidth. This architecture aims for a 20x rack-scale energy efficiency gain by 2030. The Helios rack will combine MI350 GPUs, EPYC CPUs, and Pensando NICs to house 72 GPUs with 260 TB/s of internal interconnect bandwidth. Both Dell and HPE offer high-performance, dense rack solutions for the MI350 series. However, ROCm installation demands more technical expertise than the NVIDIA CUDA setup. The ROCm software stack requires specific driver versions and manual dependency resolution that can intimidate new users. The MI355X is the better choice for cost-sensitive training workloads where users run standard PyTorch, vLLM, or SGLang.