How to Benchmark an FPGA Accelerator on Performance, Power and Accuracy

To benchmark an FPGA accelerator fairly, measure three things on the same realistic workload: performance (throughput, latency and speedup over a baseline), power and energy (measured at the board or wall, not only estimated), and accuracy (error against a trusted reference). Report them together, because an FPGA design can trade one against the others, for example by using lower-precision arithmetic to gain speed and save power at some cost in accuracy. Above all, define the baseline, the data set and what is included in the timing and power boundary, so others can reproduce the result.

FPGA accelerators are popular for applications that need high performance, low power and good accuracy. Evaluating and benchmarking them is not trivial, as there are many factors and metrics to consider. This post covers the challenges and best practices, with examples of how to measure each metric.

Key takeaways

• State the measurement boundary: kernel only, kernel plus host-device transfers, or the full application.
• Report throughput and latency separately. Batching raises throughput but usually raises latency too.
• Performance per watt (operations per second per watt) and energy per operation (operations per joule) are the same quantity in different units.
• Measure power on hardware where possible. Tool estimates are useful during design but are not a substitute.
• Accuracy matters most when the FPGA uses fixed-point or reduced-precision arithmetic. Compare against a double-precision or framework reference.

Performance

Performance is one of the most important metrics for FPGA accelerators, as it shows how fast the accelerator can process input data and produce results. It can be measured in different ways, depending on the application domain and the accelerator design. Common performance metrics are:

• Throughput is the amount of data or work processed per unit time, usually expressed in megabytes per second (MB/s), gigabytes per second (GB/s), operations per second (OPS), or a domain unit such as frames or inferences per second.
• Latency is the time elapsed between the start and the end of a computation, usually expressed in milliseconds (ms), microseconds (µs) or nanoseconds (ns). Report percentiles (for example median and 99th percentile), not only the average.
• Speedup is the ratio of the execution time of a baseline system (such as a CPU or a GPU) to the execution time of the FPGA accelerator, a dimensionless number.

To measure performance, use realistic and representative input data sets that reflect the target application. Consider the overheads and bottlenecks that can affect performance, such as data transfer over PCIe, memory access, host communication and synchronisation. Compare the FPGA accelerator with competing solutions, such as CPUs, GPUs and ASICs, using fair and consistent benchmarks.

Where performance is lost

A useful sanity check is the roofline idea: a kernel is limited either by compute (how many operations the fabric and DSP blocks can do per second) or by memory bandwidth (how fast data can be fed in). If the measured throughput is far below both limits, the problem is usually in the pipeline (stalls, low initiation interval efficiency, low clock frequency) or in the host interface.

Power

Power is another important metric, as it shows how much energy the accelerator uses during operation. It can be measured in different ways, depending on the power source and the measurement equipment. Common power and energy metrics are:

• Power consumption is the rate at which the accelerator uses electrical energy, expressed in watts (W) or milliwatts (mW). Distinguish static (idle) power from dynamic power under load.
• Power efficiency is performance divided by power, expressed in operations per second per watt (OPS/W). Because one watt is one joule per second, OPS/W is numerically the same as operations per joule (OP/J). Operations per kilowatt-hour (OP/kWh) is the same idea in a larger energy unit.
• Energy consumption is the electrical energy used for a given task, expressed in joules (J) or kilojoules (kJ). It equals average power multiplied by run time.
• Energy efficiency is the work done per unit of energy, expressed in operations per joule (OP/J) or operations per kilojoule (OP/kJ). Its inverse, energy per operation (for example nanojoules per inference), is often easier to compare.

To measure power, use accurate and reliable equipment that can capture the dynamic power variations of the accelerator. On many accelerator cards, on-board current and voltage sensors can be read through the vendor's management tools. An external power meter at the card's supply or at the wall gives a system-level figure. Consider the power of the entire system, including the host, memory subsystem and cooling, and compare against competing solutions using the same boundary.

Worked example: performance per watt

The numbers below are illustrative, for an image-classification workload.

Platform	Throughput (images/s)	Average power (W)	Images per joule	Energy per image (mJ)
CPU baseline	400	120	3.3	300
FPGA accelerator	2,000	40	50	20

Speedup is 2,000 / 400 = 5. Energy efficiency is 2,000 / 40 = 50 images per joule for the FPGA against 400 / 120, about 3.3, for the CPU, a 15 times improvement. If the FPGA figure covered only the card but the CPU figure covered the whole server, the comparison would be unfair, which is why the boundary must be stated.

Accuracy

Accuracy is another important metric, especially for numerical computation or machine learning. It can be measured in different ways, depending on the application and the design. Common accuracy metrics are:

• Error rate is the percentage of incorrect results compared with a reference solution, expressed as a percentage (%). For classifiers, top-1 and top-5 accuracy are the usual forms.
• Mean squared error (MSE) is the average of the squared differences between the accelerator's results and a reference. It is in the squared units of the output, and becomes dimensionless only if the data are normalised.
• Mean absolute error (MAE) is the average of the absolute differences between the accelerator's results and a reference, in the same units as the output.
• Root mean squared error (RMSE) is the square root of the MSE, again in the same units as the output.
• Peak signal-to-noise ratio (PSNR) compares the maximum possible signal value with the error between the accelerator's results and a reference, expressed in decibels (dB). It is calculated as 10 log₁₀(MAX² / MSE).

To measure accuracy, use high-quality reference solutions, such as a double-precision software model or the original framework model. Consider the trade-offs between accuracy and the other metrics: higher accuracy may need wider data paths, more DSP blocks and more memory bandwidth. Compare accuracy with competing solutions using fair and consistent benchmarks.

Worked example: PSNR for an 8-bit image filter

Suppose a fixed-point FPGA filter produces 8-bit output pixels (MAX = 255) and the MSE against a floating-point reference is 4.0. PSNR = 10 log₁₀(255² / 4) = 10 log₁₀(16,256) which is about 42.1 dB. RMSE is 2.0 grey levels. These numbers are illustrative.

A simple benchmarking checklist

1. Fix the workload, data set, batch size and numeric precision, and record them.
2. Choose a baseline that is well optimised, not a naive single-threaded version.
3. Define the timing boundary and the power boundary, and use the same ones for every platform.
4. Warm up, then run long enough for stable results; repeat runs and report the spread.
5. Record board, FPGA part, clock frequency, tool version and resource use (LUTs, DSPs, BRAM).
6. Report performance, power, energy and accuracy together in one table.

Common mistakes

• Quoting kernel-only time while ignoring data transfer to and from the card.
• Comparing against an unoptimised CPU or GPU baseline.
• Using tool power estimates as if they were measurements.
• Reporting peak throughput with large batches alongside latency measured at batch size one, as if both held at once.
• Changing precision on the FPGA without reporting the accuracy impact.

FAQ

Are there standard benchmarks for accelerators?

For machine learning, MLCommons publishes the MLPerf suites, which define workloads, accuracy targets and run rules, and include a power measurement methodology. For other domains, teams usually define an application-level benchmark of their own and publish its rules.

Should I report OPS/W or energy per operation?

Either, since one is the inverse of the other. Energy per operation is often easier to read and to add up across a pipeline.

How long should a power measurement run?

Long enough to cover several complete passes of the workload at steady temperature, so that start-up spikes and thermal drift do not dominate the average.