VRAM Waste and Routing Failures in MI300X MoE Fine-Tuning

Engineers provisioning AMD Instinct MI300X clusters often miscalculate memory requirements by using inference-only VRAM sizing. The MI300X contains 192 GB of HBM3 memory. MoE model training requires additional space for gradient buffers and optimizer states beyond the weight footprint. For a rank-32 adapter on a 235B Qwen3 model, the optimizer state adds approximately 600 MB to the memory footprint. Active parameter count does not dictate memory planning for MoE architectures. The base model weight footprint for a 235B MoE at FP8 equals 235 GB before adding any training overhead. If you use inference sizing for training, the job will fail.

Larger models require even more careful provisioning. A rank-32 LoRA for Qwen3-235B at FP8 fits on 2x H200 SXM5 nodes with 282 GB of capacity. DeepSeek V4-Pro requires one 8x H200 node with 1,128 GB of VRAM to hold its 865 GB of base weights. This leaves 250 GB for adapters, optimizer states, and activations. A 4-bit BitsAndBytes load for V4-Pro requires 960 GB, which makes it larger than the native checkpoint. For GLM-5.2 at INT4, the weights alone need 375 GB. Because the backward pass through a frozen base requires holding or recomputing activations, engineers must budget at least 6x H200 GPUs for GLM-5.2 QLoRA. The DeepSeek V4-Flash model weights are approximately 160 GB, which allows it to fit on 2x H200. A 72-hour Qwen3-235B run on 8x H200 costs $2,615 at 2026 rates.

Routing and Parallelism Errors

Engineers who apply ZeRO-3 to MoE models in multi-node clusters cause immediate failures. DeepSpeed ZeRO-3 shards all parameters across data-parallel ranks, which conflicts with expert parallelism where each GPU must own a complete expert set. Engineers should use ZeRO-2 to shard only gradients and optimizer states. Naive SFT on a 685B+ MoE without expert parallelism hits OOM. Engineers who ignore the auxiliary load-balancing loss cause dead-expert collapse. A coefficient of 0.01 remains the standard starting point to prevent experts from losing 10-30% of capacity by step 1000.

Engineers who apply vanilla LoRA without accounting for the router cause dead-expert collapse or load imbalance after a few hundred steps because the input distribution shifts away from the base model’s specific learned distribution. Router instability occurs when the gate network weights shift without z-loss stabilization, making the softmax output peaky. This effectively converts the MoE into a very small dense model. Recovery from this state requires resetting the router weights and restarting fine-tuning from a checkpoint. One expert might receive all tokens in a batch if the capacity factor is not tuned correctly, which causes dropped tokens and corrupts training runs silently. Expert parallelism works well when the total expert weight footprint fits across the GPU cluster.

Tensor parallelism splits individual expert weight matrices across GPUs. Use it when individual expert layers are too large for a single GPU’s VRAM.

Sequence Management

Traditional padding in variable-length training sequences wastes 60% of computation and memory. The THD format reduces memory usage by 40-60% and improves throughput by 1.5-2x. This method concatenates sequences along the token dimension. Dynamic Context Parallelism handles the remaining imbalance by selecting the CP size on a per-microbatch basis.

The THD layout collapses batch and sequence dimensions into a token dimension. Cumulative sequence length tracking marks the start and end positions of each individual sequence within the packed tensor. These parameters propagate through the attention mechanism to fused kernels to ensure queries and keys from different sequences do not attend to each other. The Dynamic-CP solver selects an effective CP size for each microbatch to avoid over-sharding short packed samples. This process avoids the overhead of re-constructing communication groups during training. How does one determine the optimal capacity factor for extremely long context runs?