Inference vs Training Silicon: Why They Are Parting Ways

In April 2026 Google announced its eighth-generation TPU as two chips instead of one. TPU 8t, for large-scale training, has 216 GB of HBM and 128 MB of on-chip SRAM. TPU 8i, for serving and post-training, has 288 GB of HBM and 384 MB of SRAM, with a lower peak compute rating of 10.1 PFLOPs against 12.6. These are Google's figures, and they show the shift plainly: the two jobs no longer want the same balance of memory and compute.

This post compares what each job asks of memory, networking and cooling. The split is sharpest in memory, real but still moving in networking, and weaker than the headlines suggest in cooling. A worked example with illustrative numbers shows how to test each point on your own design.

Two jobs with different scorecards

Training is one very large job. Each iteration has a forward pass, a backward pass and an optimiser step, and Meta's SIGCOMM 2024 paper describes GPUs synchronising gradients or parameters several times within each iteration, moving large amounts of data within milliseconds. Every accelerator waits for the slowest. MLPerf Training reflects this by scoring the time taken to reach a required model quality.

Inference is a stream of separate requests with a clock on each one. MLPerf Inference set its Llama 2 70B server limits at 2 s to first token and 200 ms per later token. Its interactive version, added in v5.0, tightens those to 450 ms and 40 ms at the 99th percentile.

Inference also splits within itself. The Splitwise paper (ISCA 2024) separates each request into a compute-intensive prompt phase and a token generation phase that leans on memory bandwidth and capacity. NVIDIA's glossary uses similar terms: prefill is compute-bound and decode is memory bandwidth-bound. The Splitwise authors report 1.4 times the throughput at 20% lower cost from running the phases on separate machines. That is why "inference silicon" is now several designs rather than one.

Side by side

This table is our reading of the published work. Treat each row as a tendency rather than a law.

	Training	Inference
Memory priority	Large shared capacity and steady throughput across a huge pod	Bandwidth per token for decode, with capacity for weights and KV cache
Network traffic	Synchronised collectives with few, large flows	Request-level traffic, increasingly with state moved between phases
Cost of one slow link	Can stall the whole job	Usually hurts the requests on that path
Power behaviour	High peak, with coordinated swings every few seconds	Lower peak, with a daily pattern
Cooling	Liquid at pod scale	Liquid at pod scale, air still possible at card scale

The next table lists real parts. These are company figures, with different dates, precisions and stages of availability, so they do not rank the parts.

Part	Aimed at	Memory	Network	Cooling and power
Google TPU 8t	Large-scale pre-training	216 GB HBM at 6,528 GB/s; 128 MB SRAM	3D torus; 9,600 chips per superpod	Liquid
Google TPU 8i	Post-training, serving, reasoning	288 GB HBM at 8,601 GB/s; 384 MB SRAM	Boardfly; up to 1,152 chips connected, up to 1,024 active	Liquid
NVIDIA Rubin GPU (Vera Rubin NVL72)	Training and inference	288 GB HBM4, 19.2 TB/s	NVLink 6 at 3 TB/s per GPU; 72 GPUs per rack	Direct liquid, 45 °C supply
Microsoft Maia 200	Inference	216 GB HBM3e at 7 TB/s; 272 MB SRAM	Two-tier Ethernet; 2.8 TB/s per chip; up to 6,144 chips	Closed-loop liquid; 750 W
NVIDIA Groq 3 LPU (LPX rack)	Low-latency decode	500 MB SRAM at 150 TB/s	2.5 TB/s chip to chip; 256 chips per rack	Liquid trays
Intel GPU D-series (Crescent Island)	Inference	Up to 480 GB LPDDR5X; Intel's own card has 160 GB	PCIe 5.0 card	Air; 350 W
Meta MTIA 2i	Inference for recommendation models	256 MB SRAM at 2.7 TB/s; 64 to 128 GB LPDDR5 at 204 GB/s	24 chips per server	85 W TDP, 65 W typical

Memory: the clearest break

Google's numbers allow a direct comparison because both chips come from one generation and one company. Dividing by peak FP4 compute, TPU 8t has about 17.1 GB of HBM and 518 GB/s of HBM bandwidth per PFLOP. TPU 8i has about 28.5 GB and 852 GB/s. That is roughly 1.6 to 1.7 times more memory per unit of compute on the inference chip, plus three times the SRAM. The arithmetic is ours. Google says the extra SRAM lets a larger KV cache stay on silicon, which reduces idle core time during long-context decoding.

On the training side, scale is the theme. Google quotes two petabytes of shared HBM in a TPU 8t superpod, which agrees with 216 GB times 9,600 chips, or about 2.07 PB.

Inference designs are heading three ways at once:

• More HBM and more SRAM. TPU 8i and Maia 200 keep HBM and add a sizeable SRAM tier.
• SRAM first. NVIDIA's Groq 3 LPU uses 500 MB of compiler-managed on-chip SRAM as its main working store, with 150 TB/s per chip. NVIDIA pairs it with Rubin GPUs, which keep prefill and decode attention because those parts benefit from large memory capacity.
• Cheaper, larger memory. Intel's D-series uses LPDDR5X. Meta's MTIA 2i pairs LPDDR with a large SRAM, and Meta reports 44% lower total cost of ownership than GPUs.

The cheaper route has become more plausible because models are changing shape. MLPerf Training v6.0 describes DeepSeek V3 as having 671 billion parameters, with 37 billion activated per token, which is about 5.5%. Intel's Hot Chips 2026 talk argues that capacity and bandwidth have decoupled, claiming that from Llama 2 70B to Kimi K2 the bytes held grew 7.5 times while the bytes read per token fell 4.4 times.

LPDDR still has a limit. In Meta's own evaluation of Llama2-7B on MTIA 2i, prefill met a 600 ms first-token requirement, but decode missed a 60 ms per-token requirement because of limited LPDDR bandwidth. HBM supply also bears on the choice, and our post on why HBM has become so tight covers that side.

Networking: same goal, different failure

Training networks are built around synchronisation. Meta built a dedicated backend network for distributed training, separate from the rest of the data centre network, with one 400G RDMA NIC per GPU in its H100-generation servers. It reports that training traffic has low entropy, so ordinary flow hashing distributes it poorly, and that default ECMP routing performed badly.

Google's pair shows the topology consequence. It says a 3D torus is efficient for the neighbour-to-neighbour traffic of dense training, but adds a latency cost for all-to-all patterns. TPU 8i therefore uses Boardfly, and Google claims that on a 1,024-chip pod the network diameter falls from 16 hops to seven.

Inference used to be the easy case, since requests were independent. That is changing. Splitwise moves the KV cache between prompt and token machines over the cluster's fast interconnect. NVIDIA's LPX design exchanges intermediate activations between GPUs and LPUs for each token. Submitters to MLPerf Inference v6.1 reported prefill and decode disaggregation too.

MLPerf hints at the remaining gap: 60% of systems in Training v6.0 were multi-node, while Inference v6.1 had a record 16 multi-node submissions among 120 systems, the largest using 512 accelerators. The counts are defined differently, so read only the direction. Once distances grow, the link itself becomes the constraint, as we discuss in where the AI hardware bottleneck moves after CoWoS.

Cooling: follows density, not the label

A tidy story would have hot training chips on liquid and cool inference chips on air. The published parts do not support that. TPU 8i, Maia 200 and the LPX rack are all inference hardware, and all are liquid cooled. Google's inference-branded Ironwood pod spans nearly 10 MW across 9,216 chips, a little under 1.1 kW per chip by our division. Google does not spell out what that power covers, so it is not a chip rating.

Air survives where density is lower. Intel's D-series is a 350 W air-cooled PCIe card aimed at existing racks. One MLPerf Inference v6.1 submitter describes both air-cooled and liquid-cooled servers using the same AMD MI355X GPU. So the choice follows power per rack and the building you already have.

The firmer difference is in how power behaves over time. The POLCA study, using production LLM clusters at a large cloud provider, reports peak power utilisation of 97% for training and 79% for inference. Training swings reached up to 37.5% of provisioned capacity, against up to 9% for inference, and the authors call the swings the main challenge in training. Those measurements come from 2023-era systems, so check them against your own fleet. NVIDIA's Vera Rubin design responds with about six times the local energy buffering of Blackwell Ultra, which it says smooths synchronised swings.

That is mainly a power delivery problem, and it reaches cooling through the peak the plant must handle. Other pressures apply to both camps. NVIDIA keeps a 45 °C coolant supply so that water can be cooled with ambient air. ServeTheHome's report of Google's Hot Chips 2026 talk says TPU 8t is the first Google design to liquid-cool its optics. The package matters as well, as our CoWoS and EMIB decision guide notes for heat concentrated under logic and adjacent HBM.

A worked example with three quick tests

The cards, link speeds, timings and fleet sizes below are illustrative assumptions, not data for any product. The only outside numbers are the MLPerf 40 ms limit, the DeepSeek V3 and Llama 2 70B parameter counts and the POLCA percentages.

Test 1: memory for decode

Take two imaginary cards. Card H has 288 GB at 8 TB/s. Card L has 480 GB at 1.5 TB/s. Assume one byte per parameter, weights spread evenly across cards, and cards reading in parallel. Ignore KV cache traffic and compute time, so this is a best case for reading weights only.

Illustrative case	Held	Read per token	Cards to hold weights	Total bandwidth	Read time per token
Dense 70B model, card H	70 GB	70 GB	1	8 TB/s	8.75 ms
Dense 70B model, card L	70 GB	70 GB	1	1.5 TB/s	46.67 ms
Sparse 671B model, card H	671 GB	37 GB	3	24 TB/s	1.54 ms
Sparse 671B model, card L	671 GB	37 GB	2	3 TB/s	12.33 ms

Against a 40 ms per-token limit, card L fails on the dense model and passes on the sparse one in about a third of the limit. Card H passes both. In this model the cheaper memory only works if your mix is sparse.

Test 2: one slow link

A training job runs on 10,000 accelerators. Each iteration takes 900 ms of compute and 100 ms of synchronisation, so 1,000 ms in total. One congested link doubles synchronisation to 200 ms, and everyone waits, so an iteration now takes 1,100 ms. Throughput falls to 1,000 / 1,100, or 90.9%. That loss of 9.1% is worth about 909 accelerators.

Now use the same 10,000 accelerators as 100 independent serving replicas of 100 each, with the same fault slowing only one replica by the same proportion. That replica loses about 9.1 accelerators' worth of throughput, and the fleet loses about 0.09%. The fault costs 100 times less in this simple model. Tightly coupling the replicas through per-token exchanges narrows that gap.

Test 3: power swings

Apply the POLCA percentages to a 10 MW row. Training swings could reach 3.75 MW, which is 37.5% of 10 MW. Inference swings could reach 0.9 MW, which is 9%. That is a step about four times larger for power delivery and cooling controls to absorb. The 10 MW is our assumption, and new hardware may behave differently.

Where the split is not clean

• The general-purpose rack remains. NVIDIA describes Vera Rubin NVL72 as a workhorse for training and inference, and rates it at 35 PFLOPS for training and 50 PFLOPS for inference in NVFP4. LPX is added around that rack and does not replace it.
• Labels are loose. Google assigns post-training to TPU 8i, so its inference chip does some training-related work.
• Roadmaps are unsettled. NVIDIA's September 2025 release described Rubin CPX, with 128 GB of GDDR7 for the context phase, as expected at the end of 2026. We have not confirmed its current status, so treat it as an announcement.
• A specialised chip carries model risk. Silicon takes far longer to design than models take to change. A part tuned for today's ratio of bytes read to bytes held may fit tomorrow's models less well.

Questions to ask before choosing

1. What is your per-token latency limit, and which model family sets it? Run Test 1 with your own active parameter counts and include the KV cache.
2. How much work crosses the network for each token? Disaggregated serving can turn a loosely coupled fleet into a tightly coupled one.
3. Can the fleet share hardware across jobs? A general-purpose part may win on utilisation if your mix moves between training and serving.
4. What evidence do you have beyond launch figures? Prefer MLPerf results for your scenario where a part has them.

Our SoC lifecycle on The Semiconductor Hub shows where such decisions fall in a programme.

Sources

• Google, "Our eighth generation TPUs: two chips for the agentic era", April 2026, and Google Cloud, "TPU 8t and TPU 8i technical deep dive", 23 April 2026.
• Google, "Ironwood: The first Google TPU for the age of inference", 9 April 2025.
• NVIDIA, "Inside the NVIDIA Vera Rubin Platform: Six New Chips, One AI Supercomputer", 5 January 2026, and Vera Rubin NVL72 product page, read 7 November 2026, for memory and NVLink bandwidth.
• NVIDIA, "Inside NVIDIA Groq 3 LPX" technical blog, 16 March 2026.
• NVIDIA, "NVIDIA Unveils Rubin CPX: A New Class of GPU Designed for Massive-Context Inference", 9 September 2025, and glossary entry "What Is Disaggregated Serving?", undated.
• Microsoft, "Maia 200: The AI accelerator built for inference", 26 January 2026.
• Intel, "Crescent Island: GPU Designed for Agentic AI Inference", Hot Chips, August 2026, and Intel GPU D-series product page, undated.
• Coburn, Tang et al. (Meta), "Meta's Second Generation AI Chip: Model-Chip Co-Design and Productionization Experiences", ISCA 2025.
• Gangidi et al. (Meta), "RDMA over Ethernet for Distributed AI Training at Meta Scale", ACM SIGCOMM 2024.
• Patel et al., "Splitwise: Efficient Generative LLM Inference Using Phase Splitting", ISCA 2024, arXiv:2311.18677.
• Patel et al., "POLCA: Power Oversubscription in LLM Cloud Providers", arXiv:2308.12908, August 2023.
• MLCommons, "MLPerf Inference v5.0 Advances Language Model Capabilities for GenAI", 2 April 2025.
• MLCommons, MLPerf Training benchmark page, undated, and "MLPerf Training v6.0" results release, 16 June 2026.
• MLCommons, "Where the Industry Is Investing: A Look at MLPerf Inference v6.1", 17 September 2026, and MLPerf Inference v6.1 supplemental discussion, September 2026 (submitters' own statements).
• ServeTheHome, "Google's TPUv8s for Training and Inference at Hot Chips 2026", 25 August 2026.

The worked example uses illustrative numbers apart from the outside figures it names. Vendor figures are the companies' own claims and targets, as published on the dates shown. This post is technical commentary and is not investment advice.