FPGAs suit AI work where latency, power or an unusual data path matter more than raw peak throughput: the hardware can be shaped to the exact precision, memory layout and pipeline a model needs. The catch has always been programming effort, so vendors built frameworks that take a trained TensorFlow, PyTorch or ONNX model and map it onto a ready-made neural processing engine inside the FPGA. Today the two main flows are AMD Vitis AI (from the former Xilinx) and the Altera FPGA AI Suite (formerly Intel), with Microsoft's Project Brainwave as the best-known example of FPGAs serving AI at cloud scale.
Key takeaways
- FPGAs win on deterministic low latency, small batch sizes, custom precision (INT8, INT4, block floating point) and close coupling to sensors and networks.
- GPUs still lead on large-batch training and on ease of programming; FPGAs are mostly used for inference.
- Vitis AI and the FPGA AI Suite both quantise a trained model and compile it for a soft or hardened AI engine, so most users never write RTL.
- OpenVINO no longer targets FPGAs directly: FPGA support ended with the 2020.3.2 LTS release. Intel's FPGA route is now the FPGA AI Suite, which uses OpenVINO as its front end.
- Corporate names changed after this post was first written: Xilinx is part of AMD, and Altera is again a separate company, majority owned by Silver Lake since September 2025.
Why AI needs specialised hardware
Artificial intelligence (AI) is the field of computer science that aims to create machines and systems that can perform tasks that normally need human intelligence, such as perception, reasoning, learning, decision making and natural language processing. AI applications are widespread and diverse, ranging from self-driving cars and smart assistants to medical diagnosis and drug discovery.
AI also poses significant challenges in computational complexity, power consumption and scalability. Traditional CPU-based architectures are limited by their largely sequential processing, high latency per operation and low energy efficiency on dense maths. GPU-based architectures offer far more parallelism and performance, but they have a fixed hardware design, high cost and high power consumption, and they reach their best efficiency only with large batches.
Field-programmable gate arrays (FPGAs) are reconfigurable hardware that can be programmed to implement almost any logic function at the circuit level. For AI they offer three advantages: flexibility, performance and efficiency.
Where FPGAs help: flexibility, performance and efficiency
Flexibility
FPGAs can be customised to match the requirements of different AI algorithms, such as data precision, memory hierarchy and parallelism. A design can use 8-bit integers in one layer and 4-bit in another, or a custom number format that a GPU does not support natively. FPGAs can also be reconfigured, fully or partially, to adapt to changing workloads and new model types after deployment.
Performance
FPGAs can achieve high throughput and low, predictable latency by exploiting the spatial parallelism and pipelining of hardware logic: each layer can be a stage in a pipeline that processes data as it streams in. They also include hardened blocks that suit AI, such as DSP slices for multiply-accumulate, embedded memory blocks (block RAM, UltraRAM, M20K) that keep weights next to the arithmetic, and high-speed transceivers that bring data straight in from cameras, radar or the network.
Efficiency
FPGAs can reduce power and cost by removing the overhead of general-purpose processors, such as instruction fetch, decode, scheduling and cache management. They usually run at lower clock frequencies than CPUs and GPUs (a few hundred MHz is typical for FPGA fabric), which helps energy per operation. The original version of this post also said FPGAs run at lower voltages; core voltages are broadly similar across modern devices, so treat that point as device dependent rather than a general rule.
A worked example: latency at batch size 1
The numbers below are illustrative, chosen only to show the arithmetic. Suppose a vision model needs 4 billion multiply-accumulates (GMAC) per frame. An FPGA engine with 4,096 MAC units running at 400 MHz delivers 4,096 x 400 million = about 1.64 trillion MAC per second. At an assumed 60 percent utilisation that is about 0.98 TMAC/s, so one frame takes roughly 4 / 0.98 = 4.1 ms, and the next frame can start as soon as the pipeline frees, with no need to wait and collect a batch. A GPU with ten times the peak rate may still show higher latency per frame at batch size 1 if its utilisation drops sharply on single images. That trade-off, not peak TOPS, is the usual reason to choose an FPGA.
| Factor | CPU | GPU | FPGA |
|---|---|---|---|
| Programming effort | Lowest | Low (CUDA, ROCm, frameworks) | Higher, eased by AI toolflows |
| Best batch size | Small | Large | Small, often 1 |
| Latency predictability | Moderate | Moderate | High |
| Custom precision | Limited | Fixed set of formats | Almost any |
| Typical use in AI | Light inference, control | Training, data-centre inference | Edge and real-time inference, network and sensor pipelines |
The programming problem
One of the challenges of using FPGAs for AI is the difficulty of programming them. CPUs and GPUs have well-established software tools and libraries, while hand-built FPGA designs need hardware description languages (HDLs) such as Verilog or VHDL, which are not familiar to most AI developers. FPGAs also had limited direct compatibility with popular AI frameworks such as TensorFlow or PyTorch.
To address this, several FPGA-based AI frameworks and platforms have been developed that simplify the design, programming and deployment of FPGA-based AI. Most follow the same flow:
- Train the model as usual in PyTorch, TensorFlow or another framework, often exported to ONNX.
- Quantise it, usually to INT8, with a calibration data set, and check the accuracy loss.
- Compile the quantised graph into instructions for a pre-built AI engine (a soft overlay in the fabric or a hardened AI engine array).
- Deploy with a runtime library on the host CPU or the on-chip Arm cores, which feeds data and collects results.
Frameworks and platforms
AMD (Xilinx) Vitis AI
A software platform for developing and deploying AI inference on AMD (formerly Xilinx) devices, for both edge and cloud. Vitis AI provides tools and libraries that work with TensorFlow, PyTorch and ONNX models, including a quantiser, an optimiser (pruning), a compiler and a runtime. The model runs on a deep learning processing unit (DPU) built into the fabric, which AMD has renamed the NPU in recent releases, targeting the Versal AI Edge series. Releases numbered 5.x and 6.x are listed in AMD's Vitis AI documentation at the time of this update.
Intel OpenVINO and the FPGA AI Suite (now Altera)
OpenVINO is a toolkit that accelerates AI inference on Intel devices. It supports a wide range of models and uses, such as computer vision, natural language processing and speech recognition. When this post was first written it was described as also targeting FPGAs through a model optimiser that converts models into an intermediate representation (IR). That needs a correction: OpenVINO's own FPGA plugin was dropped after the 2020.3.2 LTS release. The FPGA route is now the FPGA AI Suite, which takes the OpenVINO IR and compiles it for an inference IP core on Agilex, Arria 10 and other devices. The suite passed to Altera when it became a separate company.
Microsoft Project Brainwave
A platform that runs AI models on large-scale FPGA clusters in Microsoft's data centres. Brainwave was built for real-time inference with very low latency, for uses such as search ranking, image recognition, natural language processing and machine translation, and its distributed architecture lets FPGA resources scale on demand. Microsoft has described it as using Intel Stratix 10 FPGAs and a custom number format it calls narrow-precision floating point.
Microsoft Azure Machine Learning
Azure Machine Learning is a cloud service for building, training and deploying machine learning models. It offered FPGA deployment under the name Hardware Accelerated Models, based on Project Brainwave, with pre-trained models such as ResNet and support for real-time inference, which means models process data streams with low latency and high throughput. Microsoft's current Azure documentation focuses on GPU deployment, and the FPGA option was tied to an older SDK, so check Microsoft's documentation before planning on it.
What is the difference between Vitis AI and OpenVINO?
This was the most common question when choosing a framework. The table updates the original comparison for the current toolflows.
| Aspect | AMD Vitis AI | Intel OpenVINO / Altera FPGA AI Suite |
|---|---|---|
| Hardware support | AMD (Xilinx) adaptive SoCs, FPGAs and Alveo cards only | OpenVINO: Intel CPUs, GPUs and NPUs. FPGAs only through the FPGA AI Suite. |
| Model input | TensorFlow, PyTorch, ONNX, quantised and compiled by Vitis AI tools | OpenVINO reads ONNX, PyTorch and TensorFlow models and converts them to its IR; the FPGA AI Suite compiles that IR |
| Edge platforms | Zynq UltraScale+ MPSoC, Versal AI Edge, Kria modules | Agilex, Arria 10 and Cyclone class devices with the FPGA AI Suite IP |
| Cloud and data centre | Alveo accelerator cards | Intel Programmable Acceleration Cards (PACs) were used; these have since been discontinued |
The original post said Vitis AI integrates with TensorFlow or PyTorch directly while OpenVINO needs a conversion step. In practice both flows convert the model: Vitis AI quantises and compiles it into its own format for the DPU, and recent OpenVINO releases can read ONNX and PyTorch models directly before converting internally. The real difference is the hardware each one serves. Both support edge and cloud deployment; Vitis AI has the broader cloud card range, while Intel's FPGA AI offer has focused more on edge and embedded applications.
Common mistakes
- Comparing peak TOPS instead of latency and throughput at the batch size you will actually run.
- Skipping accuracy checks after INT8 quantisation; some models need quantisation-aware training.
- Using layers the AI engine does not support, which pushes them back to the CPU and ruins the speed-up.
- Ignoring memory bandwidth: large models that do not fit in on-chip memory become limited by external DDR or HBM.
- Choosing an FPGA for training, where GPUs and dedicated accelerators are usually the better tool.
FAQ
Are FPGAs faster than GPUs for AI?
Not in peak throughput for most models. They are often better in latency at small batch sizes, in energy per inference for a fixed task, and where AI sits in a stream of sensor or network data.
Do I need Verilog to run a neural network on an FPGA?
Not with Vitis AI or the FPGA AI Suite: you work in Python and the vendor tools. RTL skills help when you build custom pre- and post-processing or your own accelerator.
Can OpenVINO still target an FPGA?
Not on its own. Current OpenVINO releases do not include an FPGA plugin; use the FPGA AI Suite, which accepts OpenVINO models.
Which FPGAs are typical for edge AI?
AMD Zynq UltraScale+ MPSoC, Kria and Versal AI Edge parts, and Altera Agilex 5 and Arria 10 class parts, chosen by power budget, I/O and model size.
thanks for sharing
ReplyDelete