Llama 3.1 405B requires 810 GB of memory for FP16 precision, 405 GB for FP8, and 203 GB for INT4. This model takes up 231 GB on disk and requires 231 GB in RAM. The SambaNova SN40L provides 64 GB of on-chip memory per unit. Deployments must distribute the model across multiple RDUs to prevent immediate memory exhaustion. For the 70B model, memory requirements reach 140 GB for FP16, 70 GB for FP8, and 35 GB for INT4. The 8B model requires 16 GB for FP16, 8 GB for FP8, and 4 GB for INT4. Tensor parallelism reduces peak memory per GPU by the parallelism degree $T$ by splitting weight matrices. A 70B model that does not fit on a single 80GB GPU can fit on 4 GPUs if $T=4$.
| Hardware Model | Compute (FP8) | On-chip Memory | TDP | Process |
|---|---|---|---|---|
| SN40L | 1,024 TFLOPS | 64 GB | 800W | 5nm |
| SN30 | 330 TFLOPS (FP16) | 32 GB | 400W | 7nm |
Pipeline bubbles and micro-batch memory spikes
Engineers use pipeline parallelism to assign different layers to different GPUs. This strategy creates pipeline bubbles, which are idle periods where a GPU waits for the previous stage to finish. The fraction of time wasted on these bubbles is $(K-1)/(K-1+m)$, where $K$ is the number of pipeline stages and $m$ is the number of micro-batches. With $K=4$ stages and $m=16$ micro-batches, the bubble fraction is approximately 16%. If $m$ drops to 4, the bubble fraction rises to 43%. Increasing micro-batches reduces this idle time but causes memory usage to rise on the final stage. In a 4-GPU, DP2-PP2 training setup for Llama 3-8B, the memory on the last stage grows from 40.30 GB at 2 micro-batches to 65.47 GB at 16 micro-batches. This surge occurs because PyTorch caches the output of the LM head, which has a shape of $4096 \times 128256$, during the step. You should monitor the memory usage on the last pipeline stage during training runs. Will developers change how PyTorch handles the caching of these chunk outputs?
Sharding errors and parallelism efficiency
Incorrect sharding in the feedforward layer creates significant communication overhead. A transformer feed-forward layer with a hidden dimension of 8192 uses weight matrices of $[8192, 32768]$ and $[32768, 8192]$. If an engineer splits both weight matrices in the same dimension, such as both by rows, the system requires multiple all-reduce operations. Properly tailored implementations split one matrix by rows and the other by columns to ensure the module only requires a single all-reduce operation. Fixing this error improves feedforward layer performance by 30% to 50%. Tensor parallelism requires an all-reduce operation after each attention and feed-forward layer. While NVLink provides 600 GB/s bandwidth for all-reduce on a single node, InfiniBand is 10 to 30 times slower across nodes. DeepSpeed provides ZeRO optimizations to partition optimizer states, gradients, and parameters across GPUs. This partitioning removes memory duplication and allows the training of models that exceed single GPU limits. Data parallelism provides linear throughput scaling by running full copies of the model on multiple GPUs. However, each replica must hold the full model in memory. A 70B model requiring 4 GPUs for tensor parallelism would need 16 GPUs total to run 4 data-parallel replicas.
