Why poor rack planning kills H100 training uptime

A single NVIDIA H100 GPU draws 700W, and a standard AI training cluster with 256 GPUs requires 180 kilowatts of power. Most traditional data centers design for 5 to 8 kilowatts per rack, meaning they cannot support the 15 to 25 kilowatts required for a standard AI deployment. High-density AI infrastructure often demands 30 to 50 kilowatts per rack, while extreme density configurations reach 60 to 100 kilowatts or more. If you attempt to deploy H100 clusters without verifying the facility can handle these concentrated loads, you face immediate thermal shutdowns or electrical failures.

The mismatch between traditional facilities and modern requirements creates a physical bottleneck that many YC startups overlook when moving from cloud instances to bare metal. A 42U rack containing 8 GPU servers, with 4 GPUs in each, easily exceeds 25 to 30 kilowatts. This density requires specialized electrical distribution and cooling, such as liquid cooling or enhanced air cooling with hot/cold aisle containment.

Cooling failures and power path bottlenecks

Air cooling hits a physical limit at 50kW per rack. To cool a 100kW rack with air, you need 15,700 cubic feet per minute of airflow, which creates hurricane-force winds through small server intakes. I find the physics of air cooling at these scales to be an impossible hurdle for anyone not using liquid solutions. Fan power consumption scales with the cube of fan speed, so a 10% increase in airflow requires 33% more fan power. This creates an energy consumption spiral that makes high-density air cooling economically impossible.

You also face a massive risk if you only look at the building’s total megawatts. A facility might have 10 MW of total power, but if it cannot deliver that power to a specific rack through the UPS, busway, or PDU, the capacity is useless. A DGX H100 system has a maximum power of 10.2 kW, so four such systems in one rack create 40.8 kW of demand. This requires a 415V, 32A, three-phase N+1 design to maintain stability.

Liquid cooling offers a way to bypass these air-side limits. Direct-to-chip cooling uses cold plates to remove 70% to 75% of rack heat. However, you still need to account for the remaining 25% to 30% of heat from memory and auxiliary components that air must still carry. If you plan a cluster without considering these hybrid needs, you will experience hot spots and hardware throttling.

Mechanical and networking constraints

Planning a cluster by GPU count alone is a mistake that wastes capital. You must size the cluster by the fabric, storage, and software requirements. If a node fails during a month-long training run, your recovery protocol must be instantaneous to prevent losing significant expenditure on GPU cycles. This orchestration challenge is why scaling to thousands of GPUs requires meticulous network planning.

The physical weight of high-density racks also presents a constant danger. A fully loaded liquid-cooled AI rack can weigh 3,500 lbs. Most multi-story data centers cannot support these point-loads, which forces AI deployments to the ground floor. You must validate static and dynamic floor loading, equipment depth, and cable bend radius before you buy a single GPU.

Network bottlenecks can leave expensive GPUs sitting idle while they wait for data. In architectures like the GB200 NVL72, dozens of GPUs operate with such extreme interconnection that they function as one unified chip. If you ignore the specific needs of NVLink or the distance limits of InfiniBand cables, which must not exceed 50m, your performance will suffer.

I have seen teams order massive H100 clusters only to find their data center cannot support the weight or the power density of the delivery. Do you have the plumbing capacity to support a 250kW CDU if your liquid cooling system fails? Can you deliver 40kW to a single rack if your facility only provides 10kW per rack? If you do not map how failures in power, cooling, or networking affect the same equipment, your uptime will remain low.