TDEB Top Bar

Trending
Semiconductor Industry News — The Digital Electronics Blog
Follow Us

TDEB Masthead

The Semiconductor Hub

Semiconductors.  Chips and Systems.  Innovation.

Indexed by leading Search Engines, AI and referenced by Engineers worldwide 500+ Articles 10M+ Pageviews 30+ Reports 50+ Citations

The Fine Print Behind TOPS/W: A Benchmark Checklist for AI Chips

Murugavel Ganesan
by
0
Share Facebook X LinkedIn WhatsApp Email
AI accelerator chip on a test bench connected to a power analyser for efficiency measurement

TOPS/W looks like a clean way to rank AI chips: one number, higher is better. The trouble is that neither half of the ratio has an agreed definition. The operations may be 8-bit or 32-bit, counted at peak or on a real network. The watts may cover a compute core, a whole chip or a server at the wall.

This post gives you a checklist for testing any efficiency claim, and a worked example with illustrative numbers. In that example, a chip that leads by 1.67 times on TOPS/W ends up using about 2.7 times the energy per inference at full load, then draws almost level at light load. No single figure tells you which chip to buy until you have fixed the workload.

Key takeaways

  • A peak TOPS figure assumes every multiplier is busy. Google's TPU paper reports 92 TOPS at peak, yet six production workloads ran between 2.8 and 86 TOPS.
  • Compare inferences per joule on a named model, at a stated accuracy and latency limit, with a stated power boundary.
  • Off-chip memory can decide the result. Sze and co-authors warn that leaving it out can give a misleading view of system efficiency.
  • MLPerf is the best public starting point because its rules fix much of this. It still leaves gaps, such as cooling, and it cannot tell you about your own model.
  • Duty cycle can change the answer, so idle power belongs in the comparison.

What the ratio leaves out

The most useful short paper on this subject is by Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang and Joel Emer, in IEEE Solid-State Circuits Magazine in 2020. Its argument is that the number of processing elements and their peak rate give only a theoretical maximum, reached at 100% utilisation. Real throughput depends on how well a given network maps onto the hardware, and on whether memory can feed it. The authors also show why counting multiply-accumulate operations (MACs) is a weak proxy for either speed or energy: data movement tends to dominate energy, and not all operations move data the same distance.

Published silicon shows how wide the gap can be. The 2017 ISCA paper on Google's first TPU gives a peak of 92 TOPS. Measured on six production networks, the chip delivered 12.3, 9.7, 3.7, 2.8, 86.0 and 14.1 TOPS, with a mean of 21.4. That is about 3% of peak on the worst workload, about 93% on the best and about 23% on average. The same chip spans a factor of about 30. A TOPS/W figure built on the peak describes none of those workloads except, nearly, the best one.

The checklist

The middle column describes common practices in general terms. Some are perfectly reasonable when disclosed, and the problem arises only when the reader is left to assume otherwise.

QuestionWhat can flatter the numberWhat to ask for
1. What counts as an operation?Low bit widths, or an operation definition left unstatedOperation type and bit width for weights and activations
2. Peak or sustained?Multiplier count times clock, with full utilisation assumedMeasured run time on a named network
3. Which model?A small or old network that suits the architectureSeveral models, including ones close to yours, with weights and input size
4. At what accuracy?Aggressive quantisation or pruning, with the accuracy loss left unstatedAccuracy on a named data set against the reference model
5. Under what latency limit?Large batches that raise throughput and hide response timeBatch size, plus tail latency at the quoted throughput
6. What does the power figure cover?Core only, with DRAM, host, voltage regulators and fans left outA stated boundary, and off-chip traffic per inference
7. Was power measured during the same run?Throughput from one test and power from another, or a typical figure against a peak rateAverage power, measured over the quoted workload
8. Silicon or simulation?Post-synthesis estimates at a favourable cornerWhether it is measured. If not, the design stage, process corner, voltage and temperature
9. Which software build?A tuned one-off compiler flow, or an old baseline for the rivalSoftware versions, and whether you can reproduce it
10. How was it summarised?A best-case model presented as typical, or an unweighted meanPer-model results, then a mean weighted by your own mix

Items 3, 4, 8 and the batch-size part of 5 follow the reporting advice in Sze et al. Our post on how to benchmark an FPGA accelerator applies the same discipline to FPGAs.

Where the published evidence is strongest

Precision and accuracy

Bit width changes the meaning of an operation. The tutorial survey by Sze et al. (2017) reports that an 8-bit fixed-point multiply uses 15.5 times less energy than a 32-bit fixed-point multiply, and an 8-bit add uses 3.3 times less. These are older circuit figures, so take them as a guide to scale. The point is that a tera-operation at 8 bits is not the same amount of work as one at 32.

Accuracy has a price too. The MLPerf Power paper (HPCA 2025) found that datacentre systems running BERT at 99.9% of the reference accuracy were, on average, half as efficient as those at 99%. It also found that newer 8-bit floating-point support lifted the higher-accuracy results from about 50% to about 85% of the lower-accuracy efficiency on the same hardware. So a small step in the accuracy target can move the result a lot, and the size of the penalty depends on what the hardware supports. The original MLPerf Inference paper sets quality targets per model for this reason: giving up 1% of accuracy may be acceptable in one application and unacceptable in another.

Power boundary

The TPU paper is a good illustration because it reports both views openly. Against a Haswell CPU server, the TPU server came out at 17 to 34 times better performance per watt when host power was included, and 41 to 83 times when host power was subtracted. The hardware and the workloads are the same in both cases. Only the boundary differs. Each view answers a fair question, provided the reader is told which one it is.

Memory is the part most easily left out. Sze et al. say that ignoring off-chip accesses can give a misleading impression of efficiency, and ask for off-chip memory power or the data read and written off-chip per inference. MLPerf Power gives a real case: DLRM-v2 has only 1.06 times the MACs of ResNet in the paper's data, yet uses 2.3 times the energy per sample, which the authors attribute to data movement. With memory now a pinch point for AI systems, as our report on the AI memory squeeze describes, off-chip traffic also bears on what the system costs to build.

Latency

Throughput with no response-time limit can be a long way above what an application can use. In the TPU paper's MLP0 workload, with a 7 ms 99th-percentile limit, the CPU ran at 42% of its highest throughput, the GPU at 37% and the TPU at 80%. An offline result would have hidden most of that difference.

Software age

Comparing new silicon on tuned software against an old baseline can overstate a hardware gain. In the MLPerf Power data, 89% of the comparisons between identical datacentre hardware in consecutive rounds showed an efficiency gain from software, and 18% showed a gain above 50%. A result belongs to a chip and a software build together.

A worked example: two chips, one model

Every number in this example is illustrative, invented to show the method. Neither chip is a real product. Put in your own measurements. The model needs 5 billion operations per inference, and off-chip memory is assumed to cost 10 pJ per bit.

StepChip AChip B
Brochure: peak TOPS at quoted watts40 TOPS at 4 W24 TOPS at 4 W
Brochure TOPS/W106
Utilisation on this model25%, so 10 TOPS75%, so 18 TOPS
Inferences per second2,0003,600
Share usable within the latency limit60%, so 1,200 per second90%, so 3,240 per second
Chip power at that load3.0 W3.6 W
Off-chip traffic per inference20 MB, or 1.6 mJ5 MB, or 0.4 mJ
Memory power at that load1.92 W1.30 W
Host and board share2.0 W2.0 W
Total power6.92 W6.90 W
Inferences per joule173470
Energy per inference5.77 mJ2.13 mJ

Chip A leads the brochure comparison by 10 / 6 = 1.67 times. On the real model, Chip B delivers 470 / 173, or about 2.7 times as many inferences per joule.

That is not the end of the matter, because the table assumes the chip is always busy. Suppose your product needs only 300 inferences per second, and the chips idle at 0.2 W and 1.0 W, with the same 2.0 W host and board share still drawn when idle.

At 300 inferences per secondChip AChip B
Fraction of time busy300 / 1,200 = 25.0%300 / 3,240 = 9.3%
Busy contribution6.92 W x 0.25 = 1.73 W6.90 W x 0.093 = 0.64 W
Idle contribution2.2 W x 0.75 = 1.65 W3.0 W x 0.907 = 2.72 W
Average power3.38 W3.36 W

The 2.7 times gap has shrunk to less than 1%, and a small change in idle power would reverse it. At a required rate above 1,200 per second, on the other hand, one Chip A cannot meet the limit at all, so you would need a second one. The right answer depends on the workload, the latency limit and the duty cycle. Those are the things a TOPS/W figure leaves out. The choice is also tied to area and cost, as covered in our guide to PPA trade-offs.

What a helpful datasheet gives you

A headline such as "up to X TOPS at Y watts" cannot be tested on its own. A more useful document adds average power per model, with the bit width, temperature, voltage and batch size stated beside it. The table below shows the format. Every number in it is illustrative and describes no real product.

Model (test case)Frames per secondAverage powerBatchInferences per joule (division of the two)
Small classifier1,0004.0 W1250
Mid-size detector1255.0 W825
Second mid-size detector903.6 W825

Rows like these do not conflict, as they describe different conditions, and the models do very different amounts of work per frame. The format shows why a per-model figure is more useful: here one chip gives results 10 times apart depending on the network. Host power, accuracy on your own model and the compiler version would remain open questions.

What MLPerf settles, and what it does not

The MLCommons inference rules address many of the checklist items directly:

  • Accuracy. Many benchmarks set targets at a stated percentage of the reference model's accuracy, often at 99% and 99.9%.
  • Scenario. Single-stream, multistream, server and offline runs have their own metrics. Server runs use Poisson arrivals and a latency bound specific to the benchmark. Runs last at least 600 seconds.
  • Model changes. The Closed division allows calibration for quantisation, but bars retraining, pruning and caching queries or responses. The Open division allows arbitrary pre-processing, post-processing and models, so the two should not be read as equals.
  • Replication. Results that cannot be replicated are invalid, and up to two submissions are audited each round.
  • Power. The MLCommons power document measures AC power at the system level, covering the host, accelerators, memory and fans. Efficiency is then samples per joule.

The messaging guidelines are worth reading as a model for your own internal comparisons. Comparisons must identify differences in version, division, category, scenario and chip count. Submitters may not publish derived efficiency comparisons that use TDP or rated power in place of measured system power.

There are limits. The MLPerf Power paper says cooling power is not yet included. Air-cooled servers have fan power counted while liquid-cooled systems pay for cooling at the facility, so the boundary is not neutral between the two. A system-level result also includes the host and memory chosen by the submitter, so it describes that configuration and not the chip alone. And a benchmark result exists only where someone has submitted one, on models that may differ from yours.

A short procedure for your next evaluation

  1. Write down the workload first: model, input size, accuracy target, latency limit and required rate.
  2. Screen candidates with MLPerf results where they exist, noting division, category, scenario and version.
  3. Run your own model on real silicon or a vendor-supported flow, and record the software versions.
  4. Measure power at a boundary you choose, with the same boundary for every candidate, over the same run as the throughput, and record off-chip traffic and idle power.
  5. Report energy per inference at your actual duty cycle, per model, then weight by your real mix. The TPU paper notes that the geometric mean suits an unknown mix, while a weighted mean suits a known one.

Our SoC lifecycle on The Semiconductor Hub shows where decisions of this kind fall in a programme.

FAQ

Should TOPS/W be ignored altogether?

No. With a stated bit width, a named workload and a stated boundary, it can be a useful screen within one architecture family. It is weak as a comparison across architectures, and Sze et al. argue for reading it alongside the other metrics.

What if a vendor will only give a simulated number?

Treat it as a target. Sze et al. ask that simulated results state whether they come from after synthesis or after place and route, and which library corner was used. Ask for those details and plan a silicon check before you commit.

Sources

The worked example and the datasheet table use illustrative numbers and describe no real product. The checklist describes general practices and does not refer to any company.

Murugavel Ganesan
About the Author
Murugavel Ganesan
Product Director | SerDes · IVN · Edge AI · Compute | Ex Intel · Infineon

A semiconductor product leader working across product architecture, definition, strategy, and silicon and software execution, from concept through volume production. He has worked with teams across Asia, Europe and the USA, including at Intel, Infineon and ArrayComm.

His work covers ASICs, SoCs, FPGAs and chiplets for wireless, wireline, automotive, IoT and AI, including PCIe, CXL and DDR interfaces alongside RF and mixed-signal design.

Share Facebook X LinkedIn WhatsApp Email

Post a Comment

0 Comments

Your comments will be moderated before it appears here.

Post a Comment (0)

#buttons=(Ok, Go it!) #days=(20)

Our website uses cookies to enhance your experience. Learn more
Ok, Go it!