A Tensor Processing Unit (TPU) is Google's own chip for neural network maths. It began in 2015 as an inference accelerator for Google's data centres, built around a large systolic array of 8-bit multiply-accumulate units, and has since grown into a family of training and inference chips that Google connects in pods of thousands. The short video below, from Google Cloud Tech, walks through that history and the hardware; this post adds the numbers and the engineering reasoning behind each generation.
Video: Tensor Processing Units: History and hardware, Google Cloud Tech on YouTube.
Key takeaways
- The TPU is an application-specific accelerator (an ASIC), not a GPU. It trades general flexibility for very high efficiency on matrix multiplication, the operation that dominates neural networks.
- The heart of every TPU is a systolic array: a grid of multiply-accumulate (MAC) cells that pass data to their neighbours each clock, so operands are fetched from memory once and reused many times.
- TPU v1 (deployed 2015, described publicly in 2016 and 2017) did inference only, in 8-bit integers. From TPU v2 onward the chips also train models, using the bfloat16 number format that Google introduced.
- Scaling moved from the chip to the system: high-bandwidth memory (HBM) on each chip, a dedicated chip-to-chip interconnect, and later optical circuit switches that link thousands of chips into one pod.
- TPUs are not sold as chips. They are available through Google Cloud and power Google's own services.
Why Google built its own chip
Around 2013 Google estimated that if people used voice search for a few minutes a day, running the speech recognition networks on CPUs would need it to roughly double its data centre capacity. That projection, described by Norman Jouppi and colleagues in their ISCA 2017 paper In-Datacenter Performance Analysis of a Tensor Processing Unit, is what pushed Google to design a custom accelerator rather than buy more servers. The team reportedly went from project start to chips running in data centres in about 15 months.
The bet was simple. Neural network inference is mostly large matrix multiplications on numbers that do not need 32-bit floating point precision. A chip that does only that, in low precision, can pack far more arithmetic into the same area and power than a CPU or a GPU of that era.
How a TPU works: the systolic array
A CPU fetches operands from registers or cache for every operation. In a matrix multiply that means the same weights and activations are read again and again, and moving data costs much more energy than the arithmetic itself. A systolic array avoids most of that traffic.
- Weights are loaded into a two-dimensional grid of MAC cells and held there.
- Input activations flow in from one edge, one step per clock, moving cell to cell.
- Each cell multiplies the activation passing through by its stored weight and adds the result to a partial sum that flows in the other direction.
- Completed sums leave the far edge of the array and go to accumulators, then to the activation and pooling hardware.
Every operand read from memory is used by an entire row or column of cells before it leaves the array. That reuse is the reason a TPU can keep tens of thousands of multipliers busy with modest memory bandwidth, as long as the matrices are large enough to fill the array.
Worked example: TPU v1 peak throughput
TPU v1 has a 256 by 256 array, which is 65,536 8-bit MAC cells, clocked at 700 MHz. Each MAC counts as two operations (a multiply and an add).
- MACs per second: 65,536 x 700,000,000 = about 4.59 x 1013
- Operations per second: 2 x 4.59 x 1013 = about 92 x 1012, or 92 TOPS
That matches the 92 TOPS peak figure Google published. The same arithmetic also shows the catch: peak throughput needs all 65,536 cells busy every cycle. A small batch or a layer with a narrow matrix leaves much of the array idle, which is why real application throughput in the 2017 paper varied widely from model to model.
The TPU generations
The table summarises the main generations using figures Google has published. Peak numbers are per chip and use different number formats from generation to generation, so they are not a like-for-like comparison.
| Generation | Announced | Main use | Peak compute per chip (published) | Notable change |
|---|---|---|---|---|
| TPU v1 | 2016 (in service from 2015) | Inference | 92 TOPS, int8 | 256 x 256 systolic array, 28 nm, DDR3 memory, PCIe card |
| TPU v2 | 2017 | Training and inference | About 45 TFLOPS, bfloat16 | HBM, floating point, chip-to-chip links, pods of 256 chips |
| TPU v3 | 2018 | Training and inference | About 123 TFLOPS, bfloat16 | Liquid cooling, pods of 1,024 chips |
| TPU v4 | 2021 | Training and inference | About 275 TFLOPS, bfloat16 | Optical circuit switches, pods of 4,096 chips |
| TPU v5e and v5p | 2023 | v5e cost-efficient serving and training; v5p large training | v5p about 459 TFLOPS, bfloat16 | v5p pods of 8,960 chips |
Figures are as published by Google in its papers and Cloud documentation, rounded. The v2 and v3 per-chip figures are derived from published per-board numbers (four chips per board).
TPU v1: inference only
The first TPU sat on a PCIe card in a server and acted as a coprocessor. The host CPU sent it instructions; it had no instruction fetch of its own. Alongside the matrix unit it had 24 MiB of on-chip unified buffer for activations and 4 MiB of accumulators, with weights streamed from 8 GiB of off-chip DDR3. Google's paper reported it ran its production inference workloads around 15 to 30 times faster than the contemporary CPUs and GPUs it was compared with, with much higher performance per watt. Its weakness was memory bandwidth: several of the models it ran were limited by how fast weights could be read from DDR3, not by the array.
TPU v2 and v3: adding training
Training needs floating point for gradients and a far larger memory system. TPU v2 added HBM next to the die, replaced the 8-bit integer array with floating point units, and introduced links between chips so that many TPUs could work on one model. It also brought bfloat16: a 16-bit format that keeps the 8-bit exponent of 32-bit floating point but cuts the mantissa to 7 bits. Because the exponent range is the same as FP32, most models train in bfloat16 without the loss scaling tricks that FP16 needs. The format has since been adopted by Intel, Arm, AMD and NVIDIA hardware.
TPU v3 roughly doubled the compute per chip again and needed liquid cooling to do it.
TPU v4: the system becomes the product
With TPU v4, the interconnect was the headline. Google's ISCA 2023 paper on TPU v4 describes optical circuit switches that reconfigure how the 4,096 chips in a pod are wired together. That lets Google route around failed hosts and shape the network topology to the job. TPU v4 also added SparseCores, small dataflow processors for the embedding lookups used by recommendation models.
TPU v5e and v5p
In 2023 Google split the line: v5e aimed at cost per inference and mid-size training, and v5p at the largest training runs.
Update: later generations (added November 2026)
This post was first published in January 2024. Since then Google has announced two more generations. In May 2024 it announced Trillium (TPU v6e), which it said delivers about 4.7 times the peak compute per chip of TPU v5e. In April 2025 it announced Ironwood, its seventh-generation TPU and the first it described as designed mainly for inference, with a published peak of about 4,614 TFLOPS in FP8 per chip, 192 GB of HBM, and pods of up to 9,216 chips. These are vendor figures and use a different number format from the earlier rows, so treat them as indicative.
TPU vs GPU vs CPU
| Aspect | CPU | GPU | TPU |
|---|---|---|---|
| Design goal | Fast single threads, any code | Many parallel threads, graphics and general compute | Dense matrix maths for neural networks |
| Core compute structure | A few wide cores with vector units | Thousands of SIMT lanes plus tensor cores | Large systolic matrix units plus vector and scalar units |
| Programming | Any language | CUDA, ROCm, frameworks | Frameworks through the XLA compiler (JAX, TensorFlow, PyTorch/XLA) |
| Where to get it | Anywhere | Buy cards or rent in any cloud | Google Cloud and Google's own services only |
Common misconceptions
- "A TPU is a kind of GPU." It is not. It has no graphics pipeline and is not programmed with threads. It is closer to a very large, programmable matrix engine.
- "Peak TOPS tells you speed." Only if the array is kept full. Memory bandwidth, batch size and layer shapes decide how close a real model gets to peak.
- "You can buy a TPU." Google's data centre TPUs are rented through Google Cloud. The small Edge TPU in Coral boards is a different, low-power part.
FAQ
What does "tensor" mean in Tensor Processing Unit?
A tensor is a multi-dimensional array of numbers, the basic data type in neural network frameworks. The name also echoes TensorFlow, the framework Google released in 2015.
Why does a TPU use low precision?
Neural networks tolerate rounding well. An 8-bit or 16-bit multiplier is much smaller and uses much less energy than a 32-bit floating point one, so more of them fit on the chip.
Is a systolic array only used in TPUs?
No. The idea dates back to H. T. Kung and Charles Leiserson's work in the late 1970s, and matrix engines in many AI accelerators use similar dataflow arrays.
Can a TPU run code other than machine learning?
It can run numerical work expressed as large linear algebra through XLA, but it is a poor fit for branchy, irregular code. That is the CPU's job.
Sources and further reading
- N. P. Jouppi et al., In-Datacenter Performance Analysis of a Tensor Processing Unit, ISCA 2017.
- N. P. Jouppi et al., TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings, ISCA 2023.
- Google Cloud TPU documentation, system architecture pages for each TPU version.
- Google Cloud Tech, Tensor Processing Units: History and hardware, YouTube.
Your comments will be moderated before it appears here.