How to benchmark an FPGA accelerator on performance, power, and accuracy

In this blog post, lets look at some of the challenges and best practices for FPGA accelerator evaluation and benchmarking, and provide some examples of how to measure performance, power, and accuracy of FPGA accelerators.

FPGA accelerators are popular for applications that require high performance, low power, and high accuracy. However, evaluating and benchmarking FPGA accelerators is not a trivial task, as there are many factors and metrics to consider. 

On Performance:

Performance is one of the most important metrics for FPGA accelerators, as it reflects how fast the accelerator can process the input data and produce the output results. Performance can be measured in different ways, depending on the application domain and the accelerator design. Some common performance metrics are:

Throughput is the amount of data processed per unit time, usually expressed in megabytes per second (MB/s), gigabytes per second (GB/s), or operations per second (OPS).

Latency is the time elapsed between the start and the end of a computation, usually expressed in milliseconds (ms), microseconds (us), or nanoseconds (ns).

Speedup is the ratio of the execution time of a baseline system (such as a CPU or a GPU) to the execution time of the FPGA accelerator, usually expressed as a dimensionless number.

To measure performance, it is important to use realistic and representative input data sets that reflect the target application scenario. It is also important to consider the overheads and bottlenecks that may affect the performance, such as data transfer, memory access, communication, synchronization, etc. Moreover, it is important to compare the performance of the FPGA accelerator with other competing solutions, such as CPUs, GPUs, ASICs, etc., using fair and consistent benchmarks.

On Power:

Power is another important metric for FPGA accelerators, as it reflects how much energy the accelerator consumes during operation. Power can be measured in different ways, depending on the power source and the power measurement device. Some common power metrics are:

Power consumption is the amount of electrical energy consumed by the accelerator per unit time, usually expressed in watts (W) or milliwatts (mW).

Power efficiency is the ratio of the performance to the power consumption, usually expressed in operations per watt (OP/W), operations per joule (OP/J), or operations per kilowatt-hour (OP/kWh).

Energy consumption is the amount of electrical energy consumed by the accelerator for a given computation task, usually expressed in joules (J) or kilojoules (kJ).

Energy efficiency is the ratio of the performance to the energy consumption, usually expressed in operations per joule (OP/J) or operations per kilojoule (OP/kJ).

To measure power, it is important to use accurate and reliable power measurement devices that can capture the dynamic power variations of the FPGA accelerator. It is also important to consider the power consumption of the entire system, including the host device, the memory subsystem, the cooling system, etc. Moreover, it is important to compare the power of the FPGA accelerator with other competing solutions, using fair and consistent benchmarks.

On Accuracy:

Accuracy is another important metric for FPGA accelerators, especially for applications that involve numerical computations or machine learning algorithms. Accuracy can be measured in different ways, depending on the application domain and the accelerator design. Some common accuracy metrics are:

Error rate: the percentage of incorrect results produced by the accelerator compared to a reference solution, usually expressed as a percentage (%).

Mean squared error is the average of the squared differences between the results produced by the accelerator and a reference solution, usually expressed as a dimensionless number.

Mean absolute error is the average of the absolute differences between the results produced by the accelerator and a reference solution, usually expressed as a dimensionless number.

Root mean squared error is the square root of the mean squared error, usually expressed as a dimensionless number.

Peak signal-to-noise ratio is the ratio of the maximum possible signal value to the noise value caused by errors in results produced by the accelerator compared to a reference solution, usually expressed in decibels (dB).

To measure accuracy, it is important to use high-quality and precise reference solutions that can provide correct and exact results for comparison. It is also important to consider the trade-offs between accuracy and other metrics such as performance and power, as increasing accuracy may require more resources or more complex computations. Moreover, it is important to compare the accuracy of the FPGA accelerator with other competing solutions, using fair and consistent benchmarks.