How to Design and Implement an FPGA Accelerator for an ML Model

To design and implement an FPGA accelerator for a specific ML model, you define the model and its targets, profile it to find the compute and memory bottlenecks, design a parallel datapath and dataflow that fits the FPGA's DSP, memory and I/O resources, implement it in HLS or an HDL, verify it in simulation and on hardware, and then benchmark and tune it. The key to a good result is matching the hardware to the model: quantised arithmetic, data reuse in on-chip memory and a pipeline that keeps every multiplier busy. This post walks through the six steps and adds a worked sizing example. I assume you have some basic knowledge of FPGA programming and ML concepts.

Key takeaways

• Start from a measurable goal: latency, throughput, power and the target platform (cloud, edge or embedded).
• Most ML inference time goes into multiply-accumulate (MAC) operations, so size the design by MACs per second.
• Memory bandwidth, not compute, is often the real limit; keep weights and activations on chip and reuse them.
• Quantising to INT8 or lower lets one DSP block do more work and cuts memory traffic.
• HLS shortens development; HDL gives the finest control. Vendor frameworks can generate much of the design for you.

Why use an FPGA for ML

CPUs are flexible but run ML layers mostly in sequence. GPUs have huge throughput but work best on large batches. An FPGA lets you build a datapath shaped exactly for one model: custom bit widths, deep pipelines and direct connections to sensors or networks. That makes FPGAs attractive for low-latency, batch-size-one inference at the edge, in networking, and in scientific triggers, and for models whose operators change faster than an ASIC could be built.

Step 1: Define the ML model or task

Define the ML model or task that you want to accelerate. This could be a classification, regression, clustering or any other type of ML problem. Specify the input and output data formats, the performance metrics and the target platform (for example cloud, edge or embedded).

Write the targets down as numbers. For example (illustrative): "ResNet-style image classifier, 224 by 224 RGB input, INT8 weights, under 5 ms latency at batch size one, under 10 W board power, on a mid-range FPGA SoC." Every later decision is checked against this line.

Step 2: Analyse the model and find the bottlenecks

Analyse the ML model or task and identify the computational bottlenecks. This can be done by profiling the execution time and resource usage of the model on a CPU or GPU. Also consider the data movement and memory access patterns, as they can affect the performance and efficiency of the FPGA accelerator as much as the arithmetic does.

Worked example: one convolution layer

Take a 3 by 3 convolution with 64 input channels, 64 output channels and a 56 by 56 output (a typical early layer in a ResNet-style network). The numbers are illustrative but the method is general.

Quantity	Calculation	Result
MAC operations	56 × 56 × 64 × 64 × 9	about 115.6 million
Weights (INT8)	64 × 64 × 9 bytes	36,864 bytes (36 KB)
Output activations (INT8)	56 × 56 × 64 bytes	200,704 bytes (about 196 KB)
Time with 1,024 MACs at 300 MHz	115.6 M / 1,024 / 300 MHz	about 0.38 ms

The weights fit easily in on-chip block RAM, so they can be loaded once and reused for every output pixel. Each weight is used 3,136 times (56 × 56), which is the data reuse that makes this layer compute-bound rather than memory-bound. A fully connected layer, by contrast, uses each weight only once per input and is usually limited by memory bandwidth. Repeating this analysis for every layer shows where the time and the bandwidth go.

Step 3: Design the FPGA architecture and dataflow

Design the FPGA architecture and dataflow for the model. This involves choosing the appropriate FPGA resources, such as logic elements, memory blocks, DSP slices and hard IP cores, and connecting them in a way that maximises the parallelism and throughput of the accelerator. Also design the dataflow between the FPGA and the host system, for example using DMA engines over PCIe or an on-chip AXI interconnect in an FPGA SoC.

Common architecture choices

Architecture	How it works	Good for
Systolic array	A grid of MAC units passes data to neighbours each cycle	Convolutions and matrix multiplies with high reuse
Layer pipeline (dataflow)	Each layer gets its own hardware stage; data streams through	Small models, lowest latency
Overlay or soft processor	A fixed programmable engine runs many models from instructions	Changing models without rebuilding the bitstream

Precision is a major design lever. Moving from FP32 to INT8 cuts memory traffic by four times, and FPGA vendors document techniques for packing more than one low-precision multiply into a single DSP block.

The basic building block: a MAC unit

Whatever architecture you choose, it is built from multiply-accumulate units. The Verilog below is a two-stage INT8 MAC with a 32-bit accumulator; the multiply maps naturally onto a DSP slice.

// INT8 multiply-accumulate unit with a 32-bit accumulator
module mac_int8 (
  input  wire               clk,
  input  wire               clr,      // start a new dot product
  input  wire               valid,
  input  wire signed [7:0]  a,        // activation
  input  wire signed [7:0]  w,        // weight
  output reg  signed [31:0] acc
);
  reg  signed [15:0] prod;
  reg                prod_v;
  always @(posedge clk) begin
    prod   <= a * w;                  // stage 1: multiply (maps to a DSP block)
    prod_v <= valid;
    if (clr)         acc <= 32'sd0;   // stage 2: accumulate
    else if (prod_v) acc <= acc + prod;
  end
endmodule

A testbench feeding the vectors (12, 3), (−7, 5), (127, −2) and (−128, −128) produces 16,131, which matches 36 − 35 − 254 + 16,384. The 32-bit accumulator leaves plenty of headroom: even the worst-case product of 16,384 can be added tens of thousands of times without overflow.

Step 4: Implement with HLS or an HDL

Implement the accelerator using a high-level synthesis (HLS) tool or a hardware description language (HDL). HLS tools let you write the accelerator in a high-level language such as C or C++ and generate the corresponding HDL automatically. HDLs such as Verilog or VHDL let you describe the hardware behaviour and structure directly, at a lower level.

Approach	Examples	Strengths	Trade-offs
HDL	Verilog, SystemVerilog, VHDL	Full control of timing and resources	Slowest to write and change
HLS	AMD Vitis HLS, Intel HLS and oneAPI flows	Faster iteration, C++ testbenches	Needs pragmas and care to reach good results
ML-to-FPGA frameworks	AMD Vitis AI, Intel FPGA AI Suite and OpenVINO, hls4ml, FINN	Generate much of the design from a trained model	Less freedom; supported layers vary

In HLS, the important directives are pipelining loops so a new iteration starts every cycle, unrolling inner loops to create parallel MACs, and partitioning arrays so many values can be read in the same cycle.

Step 5: Test and debug

Test and debug the accelerator using simulation tools and hardware platforms. Simulation lets you verify the functionality and performance of the accelerator before deploying it on real hardware. Hardware platforms, such as development boards or cloud FPGA instances, let you test and debug the accelerator on real FPGA devices.

• Compare every layer's output bit for bit against a quantised software reference model, not the FP32 model.
• Use the vendor's on-chip logic analyser to see internal signals on the board.
• Test with real input data as well as random vectors; saturation and rounding bugs often hide in real data ranges.

Step 6: Evaluate and optimise

Evaluate and optimise the accelerator using benchmarking tools and techniques. Benchmarking measures performance and efficiency in terms of latency, throughput, power consumption and resource utilisation. Optimisation improves them by tuning parameters such as clock frequency, pipeline depth, loop unrolling and memory partitioning.

Metric	What to measure	Typical fix if it falls short
Latency	Time for one input end to end	Deeper layer pipelining, fewer host round trips
Throughput	Inferences per second	More parallel MACs, higher clock, batching
Power	Board watts, inferences per joule	Lower precision, clock gating, less off-chip traffic
Utilisation	DSP, BRAM, LUT and routing use	Rebalance parallelism between layers

Common mistakes

• Sizing the design by peak MACs and ignoring memory bandwidth.
• Quantising without checking accuracy on a validation set.
• Leaving the host interface until last; PCIe or DMA overhead can dominate small models.
• Filling the device to 95% or more and then failing timing closure.

FAQ

Is an FPGA faster than a GPU for ML?

It depends on the workload. FPGAs often win on latency and energy for small batches and custom precision; GPUs usually win on raw throughput for large batches and on ease of programming.

Do I need to know Verilog to build an FPGA ML accelerator?

Not always. HLS and ML-to-FPGA frameworks can produce working designs from C++ or a trained model, but HDL knowledge helps a lot when debugging and optimising.

What precision should I use?

INT8 is a common starting point for inference. Lower precisions such as 4-bit or binary are possible for some models, but always confirm accuracy after quantisation.

I can go into more detail on any of these steps based on interest. Leave a comment with the step you would like covered next.