Groq

Groq

Groq emerged from stealth in 2016 when Jonathan Ross, the engineer who designed the Tensor Processing Unit inside Google, walked away from Alphabet's hyperscale advantage to build a clean-sheet chip for inference. The firm is neither a foundry nor a GPU clone — it vertically integrates silicon, compiler, and cloud API from its Mountain View headquarters. The chip, the Language Processing Unit (LPU), arranges memory and compute in a deterministic linear chain, trading the general-purpose flexibility of a GPU for hard, repeatable speed on structured linear algebra. The origin story is inseparable from Ross's conviction that the original TPU's inference-first architecture was undervalued inside a search-advertising giant. Groq's strategy sits entirely on inference, not training. The LPU targets production workloads where per-token latency, variance, and total cost of ownership determine whether an AI product ships. The firm offers a fully-managed cloud API, GroqCloud, that exposes raw throughput to external developers; early adopters assemble large language models such as Llama, Mixtral, and Gemma into deterministic pipelines. The company does not build models, train foundation weights, or compete with its existing customer base. Instead it runs third-party open-source and proprietary architectures at 300+ tokens per second on Llama-3.1-70B, a throughput figure that dominates published benchmarks. Customers span end-user chatbots through enterprise RAG systems. Critically, Groq's compiler — not CUDA — maps model graphs to silicon, breaking the fatest moat in the semiconductor industry without requiring a single line of CUDA code to be rewritten. The firm has raised over $640M from investors including Tiger Global, D1 Capital, and BlackRock Private Equity Partners. In August 2024, Groq closed a $640M Series D at a $2.8B valuation, led by BlackRock and joined by Cisco Investments and Samsung Catalyst Fund. The capital covers a planned fleet expansion beyond the 1,000+ LPUs already deployed. Groq added Stuart Pann, a supply-chain veteran who previously ran Intel's foundry and supply-chain operations, as COO in 2023; his defection signals the operational complexity ahead for an inference-cloud operator buying wafers from GlobalFoundries instead of TSMC. Yann LeCun's team provides the most visible public validation point, running Meta's own flagship open-source models on Groq hardware rather than waiting on internal GPU allocations. The structural differentiator is a compiler-first hardware architecture that targets a single function — transformer inference — and is willing to sacrifice general-purpose programmability to reach it. No other inference provider, including NVIDIA's next-generation Blackwell, is publicly hitting Groq's 300+ t/s throughput on dense 70B-parameter models. This comes with an engineering cost: the LPU's SRAM-heavy design constrains parameter size per chip, requiring massive multi-chip orchestration for frontier-class models. The company frames that orchestration — real-time partitioning across racks — as the core barrier to entry, making its fleet a functional infrastructure daemon rather than a plug-compatible GPU accelerator.