FPGA based machine learning inference means running an already trained model on an FPGA, with the network's layers built as custom hardware pipelines instead of instructions on a CPU or GPU. The model is usually quantised to low-precision integers, mapped onto the FPGA's DSP blocks and on-chip memory either directly or through a programmable overlay engine, and connected to a host processor that sends inputs and collects results. The payoff is low, predictable latency and good energy efficiency, especially at batch size one, at the cost of a harder design flow.
Machine learning (ML) is a branch of artificial intelligence that enables computers to learn from data and make predictions or decisions. ML applications are becoming more widespread and diverse, ranging from image recognition, natural language processing and recommender systems to self-driving cars. However, ML models are often computationally intensive and need high-performance hardware to run efficiently.
Key takeaways
- Inference is running a trained model on new data; it has different needs from training (latency, power, cost per inference).
- FPGAs suit inference when latency must be low and fixed, when the batch size is small, or when the model uses unusual precision or structure.
- Quantisation (for example FP32 to INT8) is almost always the first step, because it cuts DSP, memory and bandwidth needs several times over.
- There are two broad styles: a dataflow design built for one model, or an overlay engine that runs many models.
- Off-chip memory bandwidth, not arithmetic, is often the real limit.
What inference needs
One of the challenges of ML is deploying the trained models on target devices, such as edge devices, embedded systems or cloud servers. This step is called ML inference: executing the model on new input data and producing the output. Inference has different requirements depending on the application, such as latency, accuracy, power consumption or scalability. A self-driving car needs a fixed answer within milliseconds for one camera frame; a photo-tagging service cares more about images per second per watt across a large batch.
Why FPGAs for ML inference
FPGAs have several advantages for ML inference, such as:
- High performance: FPGAs can exploit parallelism and pipelining to achieve high throughput and low latency. They can also support custom data types and operations optimised for ML models, such as 4-bit weights or binary networks.
- Low power: FPGAs run at low core voltage and modest clock frequencies and, depending on the device, can adjust power to the workload through clock gating, lower-voltage operating modes or powering down unused regions. They also reduce memory access and data movement by keeping weights and intermediate results in on-chip memory, and data movement is a large share of energy in ML.
- Flexibility: FPGAs can be reprogrammed to adapt to different ML models and algorithms, as well as different hardware platforms and interfaces. They also support hardware/software co-design and co-optimisation.
The comparison with other platforms is a trade-off, not a clear win:
| Criterion | CPU | GPU | FPGA | Inference ASIC / NPU |
|---|---|---|---|---|
| Latency at batch 1 | Medium | Medium to high | Low, deterministic | Low |
| Peak throughput | Low | Very high | Medium to high | High |
| Energy per inference | High | Medium | Low to medium | Lowest |
| Support for new layer types | Immediate | Fast (software) | Needs hardware work | Limited to what was built |
| Development effort | Lowest | Low | High | Very high (for the chip maker) |
Key concepts
Quantisation
Models are trained in 32-bit floating point, but inference rarely needs that precision. Converting weights and activations to 8-bit integers usually loses little accuracy, especially with quantisation-aware training, and an INT8 multiplier is far smaller than an FP32 one. On many FPGAs one DSP block can perform two INT8 multiplications per cycle by packing operands, roughly doubling throughput per DSP compared with one 16-bit or larger multiply.
Dataflow versus overlay
In a dataflow design, each layer gets its own hardware stage and data streams through them, like a factory line. It gives the lowest latency but must be rebuilt for each model, and only fits models small enough for the device. An overlay is a programmable neural-network processor (a grid of MAC units with its own instruction set) placed on the FPGA once; new models are compiled to it like software. Vendor deep learning processing units are of this kind. Overlays trade some efficiency for much faster model updates.
Memory hierarchy
Weights and feature maps live in external DDR or HBM, block RAM and UltraRAM on the chip, and registers in the datapath. Good designs tile each layer so that data loaded from external memory is reused many times on chip before being discarded.
A worked sizing example
The figures below are illustrative, chosen to show the method, not a benchmark of any device.
- Model: an image classifier needing 4 billion MACs per image (in the range of a mid-sized CNN).
- FPGA: 2,000 DSP blocks, each doing 2 INT8 MACs per cycle, at 300 MHz.
- Peak: 2,000 x 2 x 300 million = 1.2 trillion MACs per second.
- Realistic utilisation: 50 percent, after memory stalls and layers that map poorly, giving 0.6 trillion MACs per second.
- Throughput: 0.6 trillion / 4 billion = 150 images per second, or about 6.7 ms per image if processed one at a time.
Now check bandwidth. If the model has 25 million INT8 weights (25 MB) and they do not fit on chip, reading them once per image at 150 images per second needs about 3.75 GB/s for weights alone, before feature maps. That is within the reach of a single DDR4 channel, but feature maps for early layers can add much more; this is why tiling and on-chip reuse decide real performance.
What this series covers
In this series we introduce the key concepts and techniques for FPGA based ML inference:
- How to design and implement an FPGA accelerator for a specific ML model or task
- How to use high-level synthesis (HLS) tools to generate FPGA designs from C/C++ or Python. Examples include AMD Vitis HLS, the Intel HLS Compiler and oneAPI for FPGAs, and open-source flows such as hls4ml, which converts small neural networks written in Python frameworks into HLS code.
- How to integrate the FPGA accelerator with the software stack and the host system, typically over PCIe in a server or an AXI bus in an SoC FPGA, with a runtime library on the host.
- How to evaluate and benchmark the FPGA accelerator in terms of performance, power and accuracy
- How to optimise the FPGA accelerator for different objectives and constraints, such as latency, throughput, power or device cost.
For background on accelerators in general, see What is an FPGA accelerator, and for frameworks see FPGAs and AI: frameworks, architectures and applications.
Common mistakes
- Comparing peak TOPS only. Real performance depends on utilisation and memory bandwidth; ask for measured throughput on your model.
- Skipping accuracy checks after quantisation. Always measure accuracy on a validation set after converting to INT8 or lower.
- Ignoring the host. Pre-processing, data transfer over PCIe and driver overhead can dominate latency for small models.
- Choosing a dataflow design for a model that changes monthly. An overlay or a GPU may be the better fit.
FAQ
Can FPGAs be used for training as well?
It is possible and has been researched, but training needs high precision, large memory and fast-changing software, so GPUs and dedicated accelerators dominate it.
Do I need to write Verilog to deploy a model on an FPGA?
Not always. Vendor AI toolchains and overlays let you compile a model from a framework, and HLS lets you write C++. Custom high-performance designs still benefit from RTL skills.
Which models suit FPGAs best?
Small to medium CNNs, models with low-precision or sparse weights, and anything where a fixed, very low latency matters, such as in networking, trading, industrial vision and physics experiments.
We hope this post gives you a useful overview of FPGA based ML inference. The linked articles go into each step in more detail, and this post will be extended as the series grows.
Your comments will be moderated before it appears here.