The Groq LPU vs NVIDIA H200 in Enterprise Inference

I view NVIDIA’s $20 billion acquisition of Groq’s inference technology as the primary signal that the market is shifting toward specialized hardware. Groq raised $750 million in a funding round led by Disruptive, which brought its valuation to $6.9 billion. Strategic investors like Samsung, Cisco, Blackrock, Neuberger Berman, and Deutsche Telekom Capital Partners participated in the round. This capital helps Groq scale its infrastructure across North America, Europe, and the Middle East. Groq currently supports over two million developers and Fortune 500 enterprises. While NVIDIA maintains 90% market share with H100 and B200 chips, Groq provides an LPU architecture built for speed. This architecture targets low-latency LLM serving for real-time applications. The White House has emphasized exporting the "American AI Technology Stack," and Groq positions itself as a central pillar of this strategy. The startup landscape remains crowded, with UK-based Graphcore developing the Izanagi chip to use wafer-to-wafer bonding for large amounts of SRAM. France’s VSORA is deploying the Jotunn8 chip, which provides 3,200 TFLOPS of compute power and 288 GB of HBM3e memory using TSMC’s 5nm process. Other startups like Taalus and Tensordyne are creating custom hardware to keep power consumption low while boosting performance. The deal confirms that training dominance does not equal inference dominance.

Samsung 4nm production and Grok 3

Samsung Electronics manages the volume production of the Grok 3 LPX on its 4-nanometer process. The Grok 3 LPX system packages 256 LPUs into a single rack and uses high-speed SRAM instead of standard DRAM to reduce data access times and boost inference performance for agentic AI workloads like coding. Industry sources note that Grok 3 production yields reached above 80% after April. The 4nm line at Samsung’s Pyeongtaek campus runs at nearly 100% utilization. Samsung raised prices on new orders for its 4nm process by up to 15 percent in July 2026. This price increase follows strong demand from U.S. and Chinese customers. Samsung currently holds 7% of the global foundry market, while TSMC holds over 70%. Samsung’s partnership with NVIDIA includes supplying HBM4 for the Rubin GPU. Nebius plans to deploy the Grok 3 LPX in its "Nebius Token Factory" starting at the end of 2026. You know the market prefers the CUDA ecosystem, but the LPU momentum is hard to ignore.

NVIDIA benchmarks and custom silicon

NVIDIA H200 performance stays high on standard benchmarks. For Llama 2 70B, the H200 with TensorRT-LLM software delivers 43 percent higher performance in server scenarios than the H100. The H200 also achieved speedups of 2.4x and 2.9x in the GPT-J LLM test in offline and server scenarios. OpenAI’s Habanero chip, which uses Samsung-supplied HBM4, also outperforms NVIDIA’s GB300 in performance-per-watt and response speed. The Habanero chip provides 15.4TB/s of bandwidth per package. Microsoft’s Maia 200 chip uses TSMC’s 3nm process and 216GB of HBM3e memory. This 750W chip contains 272 GB of SRAM on the die and uses Tile Tensor Units for matrix multiply and Tile Vector Processors as a programmable SIMD engine. The chip uses a Tile Control Processor to run low-level code and orchestrate DMA work issuance. Intel’s Crescent Island chip uses the Xe3P microarchitecture and supports 160GB of LPDDR5X memory. This chip, which Intel worked on with the Open Compute Project, remains optimized for air-cooled enterprise servers. NVIDIA claims the Vera Rubin NVL72 delivers 30x more agentic AI inference throughput per megawatt than the GB300 NVL72. Independent MLPerf v6.1 results show the Vera Rubin NVL72 delivers 3.7x higher throughput than the GB300 on Qwen3-VL. An 8-GPU NVIDIA HGX H200 system with GPUs configured to a 700W TDP achieved a performance of 13.8 queries per second in the server scenario for the Stable Diffusion XL test. Will NVIDIA’s focus on Rubin’s memory bandwidth eventually nullify the speed advantage of dedicated LPU architectures?