The LPU architecture delivers massive speed and efficiency for AI workloads
The LPU architecture delivers massive speed and efficiency for AI workloads via its deterministic design. Jonathan Ross, who designed the original Google Tensor Processing Unit, founded Groq in 2016 alongside Douglas Wightman. The company produces the Language Processing Unit (LPU), a chip that replaced the original Tensor Streaming Processor (TSP) architecture, codenamed "Alan". The LPU uses a functionally sliced microarchitecture where memory units interleave with vector and matrix computation units. This design exploits dataflow locality in AI compute graphs to improve performance. Because the memory resides directly on the chip, the LPU provides an internal memory bandwidth of 80 TB/s, which gives it a 24x advantage over the 3.35 TB/s seen in the Nvidia H100. The LPU also avoids traditional reactive hardware components like branch predictors, arbiters, reordering buffers, and caches. Instead, the Groq compiler controls all execution to guarantee deterministic processing. This software-first approach allows the LPU to achieve 1 PetaOp/s performance on a single chip. The first generation LPU provides up to 250 trillion floating-point operations per second. This technology supports workloads like large language models, image classification, and predictive analysis. The first generation chip operates at a 900 MHz clock frequency using a 14nm process. The upcoming LPU v2 will use Samsung’s 4-nanometer process node. Roughly 60% of Groq’s 300 employees work as software engineers to support these systems. The design ensures deterministic processing, which is useful for applications where safety and accuracy are paramount.
| Specification | LPU (First Generation) |
|---|---|
| Peak Performance | 1 PetaOp/s |
| Floating-point Operations | 250 trillion FLOPS |
| Process Node | 14nm |
| Clock Frequency | 900 MHz |
| On-chip SRAM | 230 MB |
NVIDIA’s agreement with Groq creates a strategic way to maintain dominance
NVIDIA’s agreement with Groq creates a strategic way to maintain dominance in the inference market. NVIDIA signed a $20 billion non-exclusive licensing agreement with Groq in December 2025. This deal brings Ross and President Sunny Madra to NVIDIA, though Groq remains an independent company. NVIDIA intends to use Groq’s IP to build its AI Factory architecture and expand its influence in the inference market. The deal includes a broad hiring initiative to transfer several senior Groq executives to NVIDIA. You already know that NVIDIA dominates the training market with its GPUs. However, the rise of specialized inference chips creates new pressure. Groq has grown through strategic moves, including the 2022 acquisition of Maxeler Technologies and the 2024 acquisition of Definitive Intelligence. The company raised $350 million in a Series A round led by Disruptive, and later raised $650 million in June 2026. Groq also expanded its global footprint to 13 data centers across North America, Europe, the Middle East, and Asia Pacific. This expansion follows a $1.5 billion commitment from the Kingdom of Saudi Arabia for a new facility in Dammam. The company’s valuation rose from $1 billion after its 2021 Series C to $6.9 billion in 2025. While the deal brings Groq technology into the NVIDIA stack, Hedgeye Risk Management analysts argue that Groq’s chips remain unproven for large models because they have low memory capacity.
Groq’s LPU clearly delivers the low latency needed for high-frequency trading
Groq’s LPU clearly delivers the low latency needed for high-frequency trading environments. Trading firms like XTX Markets use massive amounts of compute to analyze market data and news in real-time. XTX Markets uses 25,000 AI chips to support its daily $250 billion in assets. These firms need low latency to interpret the flood of information for machine learning-driven trading strategies. Groq’s LPU uses SRAM for sequential processing, which provides a speed advantage for single-token inference. This contrasts with Nvidia’s reliance on High Bandwidth Memory (HBM) from companies like Samsung or Micron. To run a 70-billion parameter model like Llama 3, Groq must link hundreds of chips together in a rack, such as a 576-chip cluster. The LPU provides text generation speeds exceeding 1,600 tokens per second. In benchmarks, the LPU sustained 241 tokens per second for the 70B-parameter Llama 2 model. Training an AI model is a high-bandwidth, parallel task, similar to grading a thousand exams at once. In contrast, inference for Large Language Models is autoregressive, meaning the model generates one token at a time. You cannot calculate the tenth word until you have calculated the ninth. This separation of compute cores and memory creates the "Memory Wall" that limits current GPU architectures. Because moving data is more expensive than computing, Groq’s architecture targets a reduction in energy from 10 to 30 Joules per token down to 1 to 3 Joules. Will NVIDIA’s integration of LPU technology eventually stifle the independent development of such specialized inference chips?
