A Deep Dive into the Underlying Architecture of Groq's LPU

codeconfessions.substack.com · 3,800 words · saved by 1 readers

We’ve been talking about systems performance a lot lately. Few days back Groq made news for breaking LLM inference benchmarks with their language processing unit hardware (LPU). The success of LPU is due to the innovative design of its hardware architecture, combined with a powerful compiler. As we have been focusing on performance lately, I think understanding what lies underneath the LPU hardware and compiler would be very relevant to our discussion. Groq has not published any papers on the LPU itself, but in the previous years they have published two papers (see this 2020 paper and this 2022 paper) explaining the design and implementation of their tensor streaming processor (TSP), and how they built a distributed inference engine using TSPs. Although they have not officially stated this, the LPU most likely is based on this distributed system or an extension of its design. In this article we are going to break down the architecture of a TSP and its compiler, followed by that we will

Few days back Groq made news for breaking LLM inference benchmarks with their language processing unit hardware (LPU). The success of LPU is due to the innovative design of its hardware architecture, combined with a powerful compiler. Traditional processors such as the CPU and GPU have many sources of non-determinism built-in their design, such as the memory hierarchy, interrupts, context switching, dynamic scheduling of instructions etc. This leaves many things in the hands of the hardware when it comes to achieving maximum possible throughput. Groq’s LPU has been designed to eliminate…

A Deep Dive into the Underlying Architecture of Groq's LPU

related reading