Intel Gaudi 3 outpaces Nvidia H100 in specific AI workloads

Intel Gaudi 3 delivers 1.7x faster training times for large language models like Llama2-13B and GPT-3 175B compared to Nvidia H100. The architecture features 128 GB of HBM and 3.7 TB/s of bandwidth to handle massive AI workloads.

Intel Gaudi 3 outpaces Nvidia H100 in specific AI workloads

Performance and architecture

Intel Gaudi 3 is a competitor to Nvidia and AMD in the AI accelerator market. The chip delivers 1.7x faster training times for large language models like Llama2 – 13B and GPT-3 175B when compared to Nvidia’s H100. The hardware provides 50% faster inference throughput and 40% greater inference power-efficiency across Llama 7B and 70B parameters, as well as Falcon 180B parameter models, relative to the Nvidia H100. This chip also provides 30% faster inferencing on Llama 7B and 70B parameters, and Falcon 180B parameter models against the Nvidia H200. The performance of Gaudi 3 is obvious against H100 or H200 for higher workloads such as 70B or 180B parameter models. The hardware architecture includes 64 Tensor cores and 8 Matrix Math Engines, which doubles the 24 Tensor cores and 2 Matrix Math Engines found in the previous Gaudi 2 model. Each of the 4 DCORE units contains 16 TPCs and 24 MB of L2 cache, and each of the 2 Matrix Multiplication Engines per DCORE can take 64k parallel operations. The system provides 4x more computing power and 1.5x higher HBM memory bandwidth to tackle the growing size of Large Language Models. This higher throughput occurs because the architecture allows for a higher batch size and better management during back-propagation.

Scaling and competition

The Gaudi 3 carries 128 GB of HBM, which provides 3.7 TB/s of bandwidth. This is a significant increase over the 64 GB capacity and 2.4 TB/s bandwidth of the Gaudi 2. You should look at the memory capacity if you plan on running massive models. The chip provides 2x the network bandwidth and 1.5x higher HBM memory bandwidth compared to the previous generation. Intel Gaudi 3 uses 24x 200Gb Ethernet ports per unit to build scalable racks and clusters with thousands of units. This allows organizations to build powerful clusters that compete with Nvidia AI factory solutions. The chip can be used as a standalone card or in an eight-chip package on a single motherboard. However, the AMD MI300X delivers 2.4 times more memory capacity and 1.6 times higher memory bandwidth than the Nvidia H100. The AMD MI300X uses a hybrid 5nm and 6nm process technology and has up to 153 billion transistors. The AMD chip’s HBM3 capacity is 50% higher than the previous MI250X. Additionally, the AMD chip has a 256MB Infinity Cache, while the Nvidia H100 PCIe card only has 80GB of HBM2E. The AMD MI300X cache bandwidth is 1.6 times that of H100 at the L1 level, 3.49 times at the L2 level and 3.12 times at the last level cache. In 1v1 comparison tests, the performance of the AMD MI300X can lead the H100 by up to 20% for Llama 2 70B and FlashAttention 2, and by up to 60% for Bloom 176B in 8v8 server tests. Since the Nvidia Blackwell B200 is not released yet, a direct comparison with Gaudi 3 is not possible.

Specification Intel Gaudi 3 Gaudi 2
Tensor Cores 64 24
Matrix Math Engines 8 2
Memory Capacity 128 GB 64 GB
Memory Bandwidth 3.7 TB/s 2.4 TB/s

Software and implementation

The Intel Gaudi software stack integrates with PyTorch and DeepSpeed to streamline development. For Transformer-based models, developers use FusedSDPA, which includes a Recompute mode to reduce memory usage by recomputing parts of the forward pass during the backward pass. The Non-recompute mode avoids these recomputations but requires more memory. Large batch sizes can improve throughput, but the 128 GB HBM limit restricts this. To achieve optimal performance on HPU, developers avoid executing operations on the CPU. Developers can also use the mark_step function to avoid Out of Memory issues by breaking the graph, or to reduce recompilations caused by dynamic shapes. The software stack also provides its own implementation of PyTorch ops customized for Gaudi. Users can also enable fast Softmax to speed up execution, though this may affect inference accuracy. The Intel Neural Compressor allows for running inference with FP8 data types.

Will Intel maintain this performance edge as new competitors enter the market?

airtrain.ai
airtrain.ai

The airtrain.ai newsroom covers AI research, models and the tools built on them.

More on this topic

Stay ahead of AI

Get the week's most important AI stories delivered to your inbox every Monday.

No spam. Unsubscribe anytime.

More Stories