TDEB Top Bar

Trending
Semiconductor Industry News — The Digital Electronics Blog
Follow Us

TDEB Masthead

The Semiconductor Hub

Semiconductors.  Chips and Systems.  Innovation.

Indexed by leading Search Engines, AI and referenced by Engineers worldwide 500+ Articles 10M+ Pageviews 30+ Reports 50+ Citations

Meteor Lake NPU Explained: AI Acceleration in Intel Core Ultra

Murugavel Ganesan
by
0
Share Facebook X LinkedIn WhatsApp Email

Meteor Lake, sold as Intel Core Ultra (Series 1) from December 2023, was Intel's first PC processor with a built-in neural processing unit (NPU). It spreads AI work across three engines: the CPU for short, latency-sensitive tasks, the integrated Arc GPU for heavy parallel throughput, and the NPU for sustained, low-power AI such as background blur, noise suppression and always-on assistants. In this Intel Technology "Talking Tech" video, two Intel engineers explain how the NPU works and why software and APIs matter as much as the silicon.

Key takeaways

  • The NPU is a dedicated accelerator for dense, sustained AI inference at low power, so the CPU and GPU stay free for other work.
  • Meteor Lake is a tiled (chiplet) design; the NPU sits in the SoC tile alongside media and display blocks.
  • Its compute engines combine fixed-function multiply-accumulate (MAC) arrays with programmable DSP cores.
  • Developers reach the NPU through standard APIs and frameworks, such as OpenVINO, ONNX Runtime and DirectML, rather than programming it directly.

About the video

Intel presents Meteor Lake as bringing AI to client computing at scale. In the video, Darren Crews, Senior Principal Engineer for AI software architecture, and Intel Fellow Tom "TAP" Petersen explain how Intel Core Ultra delivers AI acceleration across the CPU, GPU and the new NPU. They cover how the NPU is optimized for dense and sustained AI workloads, and why a friendly developer experience depends on focusing on software and APIs.

Two companion videos from the same channel go further: Meteor Lake Overview: In-depth with Intel Architects and Engineers (Talking Tech), and Architecture All Access: Meteor Lake, Architecture Overview, in which VP of Design Engineering Group Tim Wilson explains Meteor Lake.

Chapters

The video comes from the Intel Technology YouTube channel, which covers Intel's technologies with explainers, news and product demos. Intel's overview of its technology areas is on its technologies page, and the channel also posts on X as @IntelTech.

Why put an NPU in a PC?

AI features on a laptop tend to fall into two groups. Some are bursty and need an answer immediately, such as a quick text prediction. Others run continuously for minutes or hours, such as background blur and eye-contact correction on a video call, live captions or noise suppression. Running the continuous kind on the CPU or GPU works, but it costs battery life and steals cycles from the applications the user is actually running.

An NPU is built for exactly that continuous kind of work. It is a block of hardware that does little except the matrix multiply-accumulate operations at the heart of neural networks, with local memory to keep data close, so it delivers more inference per watt than a general-purpose core.

CPU, GPU and NPU: who does what

EngineStrengthTypical AI use on Meteor Lake
CPULowest latency, flexible, runs any codeSmall models, light or one-off inference, pre- and post-processing
GPU (integrated Arc)Highest parallel throughputLarge models, image generation, content creation in bursts
NPU (Intel AI Boost)Best efficiency for sustained inferenceVideo call effects, background AI, always-on features

Why software matters as much as silicon

A new accelerator is only useful if developers can reach it without rewriting their applications. Intel's approach is to expose the NPU through standard routes:

  • OpenVINO, Intel's toolkit for optimizing and deploying models on CPU, GPU and NPU.
  • ONNX Runtime with execution providers that target Intel hardware.
  • DirectML and Windows ML on Windows, so applications can use the NPU through Microsoft's APIs.
  • A driver stack that handles scheduling, memory and power management, so the operating system can treat the NPU like other devices.

This is the theme of the APIs and drivers chapters of the video: the hardware must be paired with stable software interfaces before independent software vendors will build on it.

Common misunderstandings

  • "The NPU replaces the GPU for AI." It does not; for large or bursty models the GPU often has more raw throughput. The NPU wins on sustained efficiency.
  • "More TOPS always means faster AI." Peak TOPS ignores memory bandwidth, supported operators and precision; two NPUs with similar TOPS can perform very differently.
  • "Applications use the NPU automatically." Only if they, or the framework they use, target it.

FAQ

What is Intel AI Boost?

Intel's brand name for the NPU in Core Ultra processors.

Which processors have the Meteor Lake NPU?

Intel Core Ultra (Series 1) mobile processors, launched in December 2023. Later generations such as Lunar Lake have larger NPUs.

Do I need special software to use the NPU?

You need an up-to-date NPU driver and an application or framework that targets it, such as OpenVINO, ONNX Runtime or a Windows feature built on DirectML.

Deep dive: the numbers and details

This part holds the NPU's internal structure, the published TOPS figures and the arithmetic behind them. Skip it if the overview is all you need.

Inside the NPU

Intel describes the Meteor Lake NPU as two neural compute engines (NCEs). Each engine pairs an inference pipeline, a large array of MAC units for the matrix and convolution work, with programmable SHAVE DSP cores for operations that do not map neatly onto the MAC array, such as activation functions and data reshaping. A DMA engine and local scratchpad memory move tensors in and out efficiently, and a memory management unit lets the NPU share system memory safely with the rest of the chip.

The NPU sits in the SoC tile of Meteor Lake's tiled design, which Intel builds with Foveros 3D packaging. Because it is outside the compute tile, the NPU can stay active while the main CPU cores power down, which helps battery life for always-on features.

Independent measurements add detail. In Intel Meteor Lake's NPU (Chips and Cheese, April 22, 2024), Chester Lam describes the block as NPU 3720, with 4,096 INT8 MACs per cycle split across the two NCEs, 2 MB of software-managed SRAM per NCE, a clock of 1.16 GHz, and FP16 running at half the INT8 rate.

Published peak figures

Intel's specification page for the Core Ultra 7 155H lists these INT8 peak values:

EnginePeak TOPS (INT8)AI frameworks Intel lists
NPU11OpenVINO, Windows ML, DirectML, ONNX Runtime
GPU18OpenVINO, Windows ML, DirectML, ONNX Runtime, WebGPU
CPUNot listed separatelyOpenVINO, Windows ML, ONNX Runtime
Overall platform33

These are vendor peak figures; real throughput depends on the model, the precision and the memory traffic.

Worked example: where a TOPS number comes from

One multiply-accumulate counts as two operations (a multiply and an add). Using the Chips and Cheese figures:

4,096 MACs per cycle x 2 operations x 1.16 x 10^9 cycles per second
  = 9.5 x 10^12 operations per second
  = 9.5 TOPS (INT8)

That is close to, but below, Intel's 11 TOPS rating. Intel's page gives the peak figure without the clock or MAC count behind it, so treat the two numbers as from different methods. FP16 at half rate gives about 4.7 TFLOPS by the same arithmetic.

Worked example: is a model compute-bound or memory-bound?

Illustrative numbers: a convolution layer needs 2 billion operations and moves 20 MB of weights and activations. At 9.5 TOPS the math alone takes 2 x 10^9 / 9.5 x 10^12 = about 0.21 ms. If the NPU can pull data from system memory at, say, 50 GB/s, moving 20 MB takes 0.4 ms. The layer is memory-bound: halving the data (for example by quantizing from FP16 to INT8 weights) helps more than extra MACs would. This is why the per-engine scratchpad SRAM and DMA engine matter as much as the MAC count.

Murugavel Ganesan
About the Author
Murugavel Ganesan
Product Director | SerDes · IVN · Edge AI · Compute | Ex Intel · Infineon

A semiconductor product leader working across product architecture, definition, strategy, and silicon and software execution, from concept through volume production. He has worked with teams across Asia, Europe and the USA, including at Intel, Infineon and ArrayComm.

His work covers ASICs, SoCs, FPGAs and chiplets for wireless, wireline, automotive, IoT and AI, including PCIe, CXL and DDR interfaces alongside RF and mixed-signal design.

Share Facebook X LinkedIn WhatsApp Email

Post a Comment

0 Comments

Your comments will be moderated before it appears here.

Post a Comment (0)

#buttons=(Ok, Go it!) #days=(20)

Our website uses cookies to enhance your experience. Learn more
Ok, Go it!