xAI’s Colossus vs Lambda Labs’ GPU cloud for YC startups

The scale of Colossus

xAI’s Colossus supercomputer in Memphis, Tennessee, operates with 230,000 GPUs, including 30,000 Nvidia GB200 units. This cluster powers the training of xAI’s Grok model. The company built the facility in 122 days, and the timeline from the first rack placement to the start of training lasted only 19 days. This infrastructure uses Nvidia Spectrum-X Ethernet networking with Spectrum SN5600 switches and BlueField-3 SuperNICs to maintain 95% data throughput. The system avoids packet loss and application latency degradation through Spectrum-X congestion control. The Spectrum SN5600 switch supports port speeds of up to 800Gb/s and uses the Spectrum-4 switch ASIC. This networking setup allows xAI to push the boundaries of training AI models at a massive scale. The supercomputer is the largest liquid-cooled AI system ever built and provides 3.6Tbps bandwidth per server. The cluster relies on 400Gbps BlueField-3 SuperNICs and a dual-cluster network design to optimize CPU-GPU workloads. While this hardware provides extreme density, the energy requirements remain massive. xAI uses gas turbines at the Memphis site to support the facility, which attracts criticism from environmental groups concerning air pollution in the Boxtown community. Elon Musk aims for a five-year goal of bringing compute power equivalent to 50 million H100 units online.

Hidden costs of cloud rental

Lambda Labs and specialized GPU clouds provide the necessary infrastructure for teams that cannot build private data centers. For an H100, Lambda Labs charges $5.54 per hour. Specialized GPU clouds provide H100 instances for $2.00 to $3.00 per hour, while marketplaces like Vast.ai or RunPod provide them for $1.49 to $2.50 per hour. You should calculate the total cost of a workload instead of looking only at the hourly GPU rate. Egress fees cost between $0.05 and $0.12 per GB, and storage for datasets or model checkpoints costs $0.02 to $0.08 per GB per month. A workload that costs $3.15 per hour in compute may reach $4.00 to $4.50 per hour after accounting for egress, storage, networking, and management fees. Some providers charge for load balancers and API management services, which add 10% to 30% to the base cost.

Small teams use the right-size strategy to reduce costs by 8% to 12% by choosing a cheaper model like the A100 or an RTX 4090 when an H100 is unnecessary. The difference between an H100 on AWS on-demand at $6.98 per hour and a marketplace spot at $1.49 per hour is $5.49 per hour, which means a single GPU running 24/7 costs an extra $4,008 every month. Reserved instances for predictable, sustained workloads offer 30% to 50% discounts over on-demand pricing, though they require one to three year commitments.

The decision for YC training runs

YC startups training models this quarter gain more from Lambda Labs and specialized marketplaces than they do from xAI’s massive scale. The Colossus cluster handles frontier model training at a scale that requires hundreds of millions in capital and massive energy footprints. A startup cannot deploy a supercomputer in 19 days or manage 400 megawatts of gas turbines. Lambda Labs and marketplaces provide the flexibility to rent specific GPUs for experimentation or small-scale fine-tuning. Teams can save 15% to 20% of the total spend by using spot instances on marketplaces for fault-tolerant workloads like model training with regular checkpoints. To save more, teams can schedule their jobs to run only during work hours to reclaim 50% of development instance spend. Containerized workloads using Docker also provide 3% to 6% savings compared to hyperscalers. Will the rapid expansion of specialized clouds eventually negate the advantage of massive, private superclusters that require hundreds of millions in capital? Lambda Labs and marketplaces clearly win for the training runs of most YC companies.