FPGA Based Video Processing Acceleration: How It Works and Where It Fits

FPGA based video processing acceleration means building the video pipeline, such as scaling, filtering, codec stages or neural network inference, directly in programmable hardware instead of running it as software on a CPU or GPU. Because pixels can be processed as they stream in, an FPGA can deliver high, deterministic throughput with latency measured in video lines rather than frames, often at lower power than a general-purpose processor for the same fixed task.

This post introduces the basic concepts, compares FPGAs with CPUs and GPUs, works through the numbers for a 4K pipeline and looks at where FPGA acceleration is used in practice.

Photo courtesy: Zhiru Zhang

Key takeaways

• Video is a streaming workload with fixed, regular operations on every pixel, which suits a hardware pipeline.
• An FPGA processes pixels as they arrive, using a few line buffers instead of whole frame buffers, so latency can be a few lines rather than one or more frames.
• Throughput comes from parallelism: several pixels per clock and many operations per pixel in the same cycle.
• FPGAs win for fixed, latency-sensitive pipelines at moderate volume. GPUs usually win for flexible, large-batch work, and dedicated ASICs win at very high volume.
• Many FPGA video products combine programmable logic with hardened blocks, such as codec units and processor cores.

FPGA vs CPU and GPU

CPUs (central processing units) and GPUs (graphics processing units) are the most common processors used for video processing. A CPU is a general-purpose processor that executes a stream of instructions. A GPU is a specialised processor that runs the same operation on large arrays of data in parallel. Both have fixed architectures designed for a broad range of applications, and both rely on software to implement the video algorithms.

An FPGA (field programmable gate array) is a programmable hardware device. It contains a matrix of configurable logic blocks and programmable interconnect that routes data between them, plus dedicated blocks such as block RAM, DSP slices for multiply-accumulate, high-speed transceivers and, in many devices, hard processor cores. Its configuration is described in an HDL such as Verilog or VHDL, or generated from C++ through high-level synthesis (HLS). The FPGA can be shaped to match the exact needs of one video pipeline, trading resources against performance and power.

	CPU	GPU	FPGA
Execution model	Instructions, a few wide cores	Instructions, thousands of lanes running in lockstep	Custom datapath, data flows through hardware
Typical latency	Frame based, milliseconds	Frame or batch based, milliseconds	Line or pixel based, microseconds possible
Determinism	Affected by OS, caches and scheduling	Affected by scheduling and memory	Cycle accurate, fixed timing
Direct video I/O	Through capture cards and drivers	Through host memory or special interfaces	Can connect directly to HDMI, SDI, MIPI or Ethernet
Ease of change	Easiest, software	Easy, software with GPU frameworks	Harder, hardware design and long compile times
Peak raw compute	Lowest	Highest for large parallel workloads	Depends on device and design

Why FPGAs suit video

1. Parallelism: an FPGA can run many processing units concurrently on different parts of the video data. It can process several pixels per clock (pixel parallelism), run every stage of the pipeline at once (pipeline parallelism) and handle several streams side by side. This increases throughput and reduces latency.
2. Streaming with small memories: most image operations only need a small neighbourhood of pixels. A 3x3 filter needs the current line and two previous lines, so the design keeps two line buffers in on-chip RAM and never waits for a whole frame to be stored.
3. Flexibility: the same device can be reconfigured for different formats, resolutions, frame rates and algorithms. Some devices support partial reconfiguration, which swaps one region of the design while the rest keeps running. New features or standards can be added with a new bitstream instead of new hardware, as long as the device has enough resources.
4. Energy efficiency: a custom datapath avoids much of the overhead of instruction fetch, decode, caches and an operating system, so for a fixed task an FPGA often uses less energy per frame than a CPU or GPU. FPGA designs can also gate clocks to unused logic. Wide voltage and frequency scaling at run time is much more limited on FPGAs than on CPUs and GPUs, so most of the saving comes from the architecture itself.

Worked example: a 4K60 filter pipeline

The numbers below are illustrative, to show how an FPGA designer sizes a pipeline. They use standard 4K60 timing.

• Active picture: 3840 x 2160 pixels at 60 frames per second, which is about 497.7 million pixels per second.
• Including blanking, standard 4K60 timing uses a total raster of 4400 x 2250, so the pixel clock is 4400 x 2250 x 60 = 594 MHz.
• 594 MHz is hard to close in FPGA fabric, so the pipeline processes 4 pixels per clock at 148.5 MHz. This is a common choice.

Now add a 3x3 convolution (for example a sharpening filter) on 8-bit RGB:

• Multiplies per pixel: 9 taps x 3 colour channels = 27.
• Multiply-accumulates per second: 27 x 497.7 million, about 13.4 billion.
• At 4 pixels per clock, the design needs 27 x 4 = 108 multipliers working every cycle. Small constant coefficients can often be built from adders and shifts, so the real DSP slice count may be lower.
• Line buffers: 2 lines x 3840 pixels x 24 bits = 184,320 bits, which is about five 36 Kb block RAMs before allowing for how the tools map width and depth.
• Added latency: about one line plus the pipeline depth. One line at this timing lasts 1 / (60 x 2250), about 7.4 microseconds. A frame-based software approach adds at least one frame, about 16.7 milliseconds.

The same arithmetic, with different numbers, sizes scalers, colour space converters, deinterlacers and the front end of neural network accelerators.

Where FPGA video acceleration is used

1. Video encoding and decoding: FPGAs accelerate compression and decompression with codecs such as H.264/AVC, H.265/HEVC and AV1. Codec engines can run several streams at once and support high resolutions (4K or 8K) and frame rates (60 or 120 fps). Many products use hardened codec blocks inside the FPGA for the core encode and decode, with programmable logic around them. For example, AMD Zynq UltraScale+ EV devices include a hard H.264/H.265 video codec unit, and the AMD (formerly Xilinx) Alveo U30 card used such devices for data centre transcoding.
2. Video analytics: FPGAs run detection, recognition and classification pipelines, including deep neural networks, on several streams in real time. Typical uses are traffic and security cameras, factory inspection and medical imaging, where low and predictable latency matters.
3. Video enhancement: FPGAs improve video quality with noise reduction, deinterlacing, scaling, sharpening, colour correction and HDR tone mapping, often in broadcast equipment, displays and professional cameras.
4. Video I/O and switching: broadcast and pro-AV equipment uses FPGAs to receive and transmit SDI, HDMI, DisplayPort and video over IP (for example SMPTE ST 2110), with processing in between.

How designs are built

A typical FPGA video design uses a streaming interface between blocks, most often AXI4-Stream in AMD devices, with a video DMA to move frames to external memory when a full frame is needed, for example for frame rate conversion. Vendors provide IP for common functions, such as scalers, colour converters, test pattern generators and video timing controllers, and libraries for HLS, such as AMD's Vitis Vision library. Custom blocks are written in RTL or in C++ for HLS. Academic research, including Zhiru Zhang's group at Cornell University, has contributed widely to HLS and heterogeneous computing for this kind of design.

Common mistakes

• Sizing the pipeline for active pixels only and forgetting blanking, or the reverse, so the clock rate or buffering is wrong.
• Using external memory frame buffers when line buffers would do, which adds latency and memory bandwidth.
• Underestimating memory bandwidth for multi-stream or full-frame algorithms; DDR or HBM bandwidth is often the real limit, not logic.
• Ignoring borders: the first and last lines and columns of a filter window need defined handling.
• Choosing an FPGA for an algorithm that changes every few weeks, where a GPU and software would be cheaper to maintain.

FAQ

Is an FPGA faster than a GPU for video?

It depends on the workload. For a fixed streaming pipeline with tight latency, an FPGA is often better in latency, determinism and energy per frame. For large batch processing or algorithms that change often, a GPU usually gives more raw throughput and easier development.

Why does an FPGA have lower latency?

It processes pixels as they arrive and needs only a few lines of buffering for most filters, so output can start microseconds after input. Software pipelines usually wait for complete frames.

Do FPGAs encode H.265 and AV1 in programmable logic?

Some IP vendors offer soft encoders, but many products use hardened codec blocks inside the FPGA or a separate codec chip, with the FPGA fabric handling pre-processing, scaling and I/O around it.

Do I need to write Verilog to use FPGA video acceleration?

Not always. Vendor IP, HLS from C++ and libraries cover many functions, but RTL skills remain useful for timing closure, custom interfaces and debug.

Conclusion

FPGA based video processing acceleration offers high throughput and low, predictable latency for video applications. It draws on parallelism, streaming with small on-chip memories, flexibility and energy efficiency to beat CPU or GPU solutions for many fixed pipelines, and it can support new features or standards by reconfiguring the hardware without changing the physical device. The trade-off is harder development, so it pays most where latency, power or direct video I/O matter.