Edge AI Chips and Software Ecosystem 2025-2026: A Detailed Analysis

Special report, February 2026, reviewed and corrected in November 2026. Market figures are as published by the sources named; most are forecasts and have not been re-verified.

Edge AI means running AI models on the device where the data is created, such as a phone, a car, a robot, a camera or a sensor, instead of sending everything to a cloud data centre. In 2025 and 2026 it moved from pilots to real deployment, driven by three things together: dedicated silicon (NPUs, edge GPUs and custom accelerators), compiler and runtime software that maps models onto that silicon, and model compression that makes large models small enough to fit. The reasons are the same as ever: latency, bandwidth, power and data privacy.

This report, which I first called "The Architecture of Intelligence", looks at the hardware, the software, the optimisation techniques and the markets that make up the edge AI ecosystem, and where the hard limits still are.

Key takeaways

• Data centre chips dominate AI revenue; edge chips dominate unit volume. NVIDIA reported USD 51.2 billion of data centre revenue in a single quarter (Q3 fiscal 2026), while edge NPUs ship in billions of phones, PCs and cars.
• The NPU is now standard in flagship phones and PCs. At the high end, NVIDIA Jetson Thor brings Blackwell-class GPU compute with FP4 support to robots.
• RISC-V is gaining ground for custom AI silicon because designers can add their own instructions and accelerators.
• Software fragmentation is the main cost. MLIR, Apache TVM and vendor runtimes such as TensorRT, OpenVINO and LiteRT are how the industry is tackling it.
• Quantisation, pruning, distillation and hardware-aware architecture search are no longer optional; most edge deployments need at least INT8 quantisation.
• Memory bandwidth (the "memory wall") and security remain the hardest open problems.

Market dynamics: why the edge matters economically

The AI chipset market is now a main driver of the semiconductor industry. One widely quoted market research estimate values the global AI chipset market at about USD 86.37 billion in 2025, rising to USD 281.57 billion by 2030, a compound annual growth rate (CAGR) of 26.66 percent. Estimates like this vary a lot between research firms, so treat it as one view rather than a consensus.

There is a clear split between data centre and edge hardware. Data centre chips are high-value, lower-volume shipments that dominate revenue today; NVIDIA's data centre revenue of USD 51.2 billion in its third quarter of fiscal 2026 (reported November 2025) shows the scale. Edge and inference chips, by contrast, drive volume across billions of devices.

Macro drivers

Demand for edge intelligence follows the volume of data produced by the Internet of Things (IoT). IDC forecast in 2019 that there would be about 41.6 billion connected devices in 2025, producing nearly 79 zettabytes of data a year. Sending all of that to central clouds is neither technically nor economically practical. Gartner has long projected that by 2025 about 75 percent of enterprise-generated data would be created and processed outside traditional centralised data centres.

Governments are backing the shift. Japan's Ministry of Economy, Trade and Industry (METI), in the 2023 revision of its semiconductor and digital industry strategy, set a goal of more than 15 trillion yen (very roughly USD 100 billion) of domestic semiconductor sales by 2030, about triple the 2020 level, supported by subsidies for leading-edge fabs. (The original version of this report gave a USD 245 billion target, which does not match METI's published goal.) Programmes like this aim to secure supply of advanced logic and high-bandwidth memory (HBM) for AI workloads.

Market metric	2025 estimate	2030 projection	Key drivers
Global AI chipset market	USD 86.37 billion	USD 281.57 billion	Generative AI, edge inference, autonomous systems
Edge AI software market	USD 2.40 billion	USD 8.89 billion (2031)	Federated learning, real-time analytics
PC unit sales	273 million	N/A	AI PC replacement cycle
Smartphone unit sales	1.24 billion	N/A	On-device generative AI, multimodal support
Custom ASICs (edge)	USD 7.8 billion	N/A	Vertical integration, power optimisation

Figures in this table are from market research estimates published in 2025 and early 2026 and have not been re-verified; different firms publish different numbers.

The "shift-left" approach, where software development and system integration start earlier in the chip design cycle using virtual platforms and emulation, has become standard practice to shorten time to market for these specialised chips. It matters because research and development spending in the chip industry keeps growing faster than earnings; one industry analysis put R&D at an estimated 52 percent of EBIT in 2024 (reported figure, not re-verified).

Semiconductor architectures: the move to specialised silicon

The defining trend of 2025 and 2026 is the move from general-purpose processing to domain-specific acceleration. Edge devices now combine CPUs with GPUs, neural processing units (NPUs) and sometimes custom accelerators on one chip.

The neural processing unit (NPU)

The NPU is the centre of edge AI silicon. It is built to run the matrix multiplications and tensor operations of deep learning in parallel, at far better energy per operation than a CPU. At the high end of the edge market, NVIDIA's move from the Ampere-based Jetson AGX Orin to the Blackwell-based Jetson AGX Thor shows the generational step in what NVIDIA calls "physical AI".

NVIDIA states that Jetson Thor, designed for advanced robotics and autonomous machines, delivers up to 2,070 FP4 TFLOPS of AI compute (a sparse figure), about 7.5 times the AI compute of AGX Orin, with about 3.5 times better energy efficiency. A key feature is native support for FP4 (4-bit floating point), managed by a Transformer Engine that switches between FP4 and FP8 to balance performance and accuracy for large language models (LLMs) and vision language models (VLMs).

Feature	NVIDIA Jetson AGX Orin	NVIDIA Jetson AGX Thor	What it means
Peak AI performance	275 TOPS (INT8, sparse)	2,070 TFLOPS (FP4, sparse)	Enables on-device foundation models
Energy efficiency	Baseline	About 3.5x higher (NVIDIA figure)	Longer run time for mobile robots
Memory capacity	64 GB LPDDR5	128 GB LPDDR5X	Supports 70B+ parameter models locally, with 4-bit or 8-bit quantisation
Memory bandwidth	204.8 GB/s	273 GB/s	Higher throughput for real-time sensing
Architecture	Ampere GPU, Arm Cortex-A78AE CPU	Blackwell GPU, Arm Neoverse-V3AE CPU	Shift to centralised domain compute

Note the comparison is not like for like: Orin's headline is INT8 TOPS and Thor's is FP4 TFLOPS. Part of the 7.5 times gain comes from the lower precision.

The automotive version of the chip, NVIDIA DRIVE Thor, is promoted as able to run ADAS, the digital cockpit and parking functions on one chip, in hardware-isolated partitions. This is meant to end the practice of stacking several separate chips and electronic control units, simplifying the wiring and the supply chain for carmakers. Reported claims of around 40 percent less wiring harness complexity come from industry marketing and depend heavily on the vehicle architecture; we have not verified them.

RISC-V: open-standard AI silicon

Arm and NVIDIA architectures dominate the market today, but RISC-V is gaining traction as an open alternative for custom AI acceleration. Because the instruction set is extensible, designers can add custom function units (CFUs) and instructions for specific machine learning workloads without asking anyone's permission.

Tenstorrent, led by veteran chip architect Jim Keller, licenses high-performance RISC-V CPU IP, including the Ascalon-X core, positioned against Arm's Neoverse V2 and V3. Tenstorrent has reported about 22 SPECint2006 per GHz for Ascalon and has worked with Samsung's SF4X process; these are company figures. Its Tensix Neo AI cores are reported to group four cores into a cluster that shares memory and network-on-chip (NoC) resources, improving area efficiency over single-core tiles.

For compact edge use, Tenstorrent and Razer have shown a modular AI accelerator that connects over Thunderbolt 5, with reports of up to four units daisy-chained to run larger models from a laptop. Esperanto Technologies built the ET-SoC-1 from more than a thousand low-power "Minion" RISC-V cores for high-throughput, low-power inference and recommendation models.

Mobile SoCs: Qualcomm, Apple and MediaTek

The smartphone is still the largest volume driver for edge AI. The 2025 flagship chips, Qualcomm's Snapdragon 8 Elite, Apple's A18 Pro and MediaTek's Dimensity 9400, show how far mobile NPU integration has come.

Snapdragon 8 Elite uses Qualcomm's custom Oryon CPU and a Hexagon NPU that Qualcomm says is 45 percent faster than its predecessor. Published Geekbench 6 results put it ahead of the A18 Pro in multi-core tests (one widely reported comparison: about 10,521 against 8,184; scores vary by device and run). Apple keeps an edge in single-core performance and CPU efficiency, and its 16-core Neural Engine is rated at 35 TOPS.

MediaTek's Dimensity 9400 focuses on on-device features such as video generation and LoRA (low-rank adaptation) fine-tuning on the handset. MediaTek says its NPU 890 gives 80 percent faster LLM prompt performance while being 35 percent more power-efficient than the previous generation.

Processor	CPU	NPU highlight	Benchmark context
Snapdragon 8 Elite	8-core Oryon (up to 4.32 GHz)	Hexagon (45% faster, vendor claim)	Strong GPU results in AnTuTu (over 1.1 million reported)
Apple A18 Pro	6-core custom (up to 4.04 GHz)	16-core Neural Engine, 35 TOPS	About 10% single-core lead (reported)
Dimensity 9400	8-core, with Arm Cortex-X925	NPU 890, on-device LoRA training	Multi-core close to Qualcomm (reported)

Software: from fragmentation to abstraction

The variety of edge hardware creates a serious software problem. Developers often have to write hardware-specific code to get peak performance, which is slow and costly. The response in 2025 and 2026 has been unified compiler infrastructure.

MLIR and modular compiler stacks

MLIR (Multi-Level Intermediate Representation), part of the LLVM project, has become a foundation for tackling this "fragmentation tax". It lets developers define domain-specific dialects that capture high-level machine learning operators and lower them step by step to machine-specific instructions.

Data, model and system optimisation increasingly run through MLIR-based tools. Operator fusion, for example merging a Conv2D and the ReLU that follows it into a single kernel, has been reported to give 1.3 to 1.5 times faster execution on Arm Cortex-M CPUs, because the intermediate result never goes back to memory. Intel has reported that automated MLIR transformations such as loop tiling and vectorisation can reach over 90 percent of the performance of hand-written kernels. Both figures are workload-dependent research results.

Apache TVM and heterogeneous mapping

Apache TVM remains an important framework for mapping models to constrained edge targets. MATCH (Model-Aware TVM-based Compilation), an academic framework, is an example of hardware and software co-design: it uses a model of each target to retarget neural networks across microcontrollers and accelerators. Its authors reported average inference latency reductions of around 60 times against plain TVM on their benchmarks, mainly by making use of on-chip accelerators that a generic flow leaves idle.

Lightweight runtimes and frameworks

Ultra-low-power TinyML depends on very small runtimes. LiteRT (formerly TensorFlow Lite) is widely used; its microcontroller edition, LiteRT for Microcontrollers (formerly TensorFlow Lite Micro), has a core runtime that Google says fits in about 16 KB on an Arm Cortex-M3, which suits wearables and wake-word detection.

Framework	Main strength	Typical use
LiteRT	Very lightweight and portable	Wearables, wake-word detection
PyTorch Mobile (now ExecuTorch)	Fast prototyping, research to device	Computer vision on Android and iOS
Edge Impulse	End-to-end TinyML workflow	Industrial IoT, predictive maintenance
OpenVINO	Optimisation for Intel hardware	Smart cameras, retail analytics
NVIDIA TensorRT	Peak performance on Jetson and Blackwell	Robotics, autonomous driving
STM32Cube.AI	Optimisation for STM32 microcontrollers	Embedded industrial sensors

"Agentic AI" is adding another software layer in 2026: multi-agent orchestration, where software agents plan, call tools and check results on their own instead of answering a single prompt. Gartner expects domain-specific language models (DSLMs) to make up the majority of generative AI models used by enterprises by 2028; coverage of Gartner's forecast quotes thresholds from more than half to more than 60 percent. Either way, software stacks will need to run specialised models close to the data.

Model optimisation: the science of compression

Fitting capable models onto resource-constrained devices takes aggressive optimisation. In 2025 compression became a baseline requirement for deployment rather than a nice-to-have.

Quantisation

Quantisation reduces the bit width of weights and activations from 32-bit floating point (FP32) to INT8, INT4 or lower. Moving from FP32 to INT8 cuts model size by four and lets hardware use cheaper integer arithmetic.

• Post-training quantisation (PTQ) converts a trained model. It is quick, but usually needs a small calibration dataset to limit accuracy loss.
• Quantisation-aware training (QAT) simulates low precision during training so the model learns to compensate. It gives noticeably better results below 8 bits.
• Mixture of formats quantisation (MoFQ) reflects research showing that layers differ in their sensitivity to precision. It picks the best format or bit width layer by layer.

Worked example: MobileNet-V2 has about 3.4 million parameters. At 4 bytes each (FP32) that is about 14 MB; at 1 byte each (INT8) it is about 3.5 MB, typically with under 1 percent accuracy loss on ImageNet. That is a good fit for a phone or a Jetson module. A small microcontroller board such as the Arduino Nano 33 BLE Sense, with 1 MB of flash and 256 KB of RAM, cannot hold even the INT8 model; vision on boards like this uses much smaller variants, such as a reduced-width MobileNet at low input resolution. (The original report said the 3.5 MB model ran on that board, which is not possible.)

Pruning and sparsity

Pruning removes redundant parameters, producing sparse models that need less memory and fewer operations.

1. Unstructured pruning zeroes individual weights. Compression on paper is high, but ordinary processors gain little because sparse matrix maths is hard to run efficiently.
2. Structured pruning removes whole channels, filters or layers. It maps cleanly onto the vector units of NPUs and GPUs. As a rough rule, pruning about half the compute of a ResNet-50 can approach twice the speed on a device such as a Jetson Nano, though real gains depend on which layers are pruned.
3. Dynamic pruning skips computation at run time depending on the input. A security camera can skip heavy feature extraction on frames with no motion, which greatly extends battery life on remote nodes.

Neural architecture search and knowledge distillation

Instead of shrinking cloud models, neural architecture search (NAS) designs models for the target hardware's memory and compute budget. MobileNetV3 was found with hardware-aware NAS for mobile phone CPUs, and MCUNet (MIT) applied NAS with a co-designed inference engine to fit ImageNet-scale models into microcontrollers with only a few hundred kilobytes of SRAM and about 1 MB of flash.

Knowledge distillation trains a compact "student" model to imitate a large "teacher". TinyBERT is a well-known example: its authors reported that a 4-layer TinyBERT kept about 96 percent of BERT-base's benchmark performance while being about 7.5 times smaller and 9.4 times faster. (The original report gave 90 percent of the accuracy at 10 percent of the latency, which understated the accuracy.)

Federated learning: distributed intelligence and privacy

The need for privacy and local adaptation is driving federated learning (FL), where devices train a shared model together without sharing raw data. It reduces the trust problem of collecting personal data and helps with regulation such as the EU's GDPR and the EU AI Act.

Frameworks for on-device training

Flower and FedML are leading open frameworks for FL at scale. Flower is framework-agnostic, supporting PyTorch, TensorFlow and Hugging Face models, and is built to cope with the variety of edge devices.

Moving from fixed inference to on-device training allows personalisation, where a model adapts to one user's voice or habits locally. Research projects such as TinyFL are exploring federated transfer learning on microcontrollers, so that even milliwatt-class devices can contribute to model updates using parameter-efficient fine-tuning (PEFT).

Hierarchical and hybrid FL

Research is moving towards hierarchical FL that follows real network structure (device, edge, cloud). Model updates are first aggregated at a local edge server, such as one next to a 5G base station, before going to the cloud, which cuts backbone traffic and latency.

FL architecture	Mechanism	Benefit and cost
Centralised FL	Star topology with a cloud aggregator	Simple coordination, but high latency
Hierarchical FL	Multi-tier aggregation at edge nodes	Reduces wide-area network bottlenecks
Decentralised FL	Peer to peer (gossip protocols)	No single point of failure; suits drone swarms
Split learning	Model split between device and edge server	Offloads heavy compute from TinyML devices

Sustainability: towards "green AI"

As AI scales, its energy use has become a strategic issue. A Lawrence Berkeley National Laboratory report (December 2024) estimated that US data centres used about 4.4 percent of the country's electricity in 2023, with sharp growth expected. Edge AI can help by processing data locally on very low-power silicon, avoiding some data transmission and data centre cooling load.

"Green federated learning", which schedules training rounds for times when renewable power is available on the local grid, is an active research theme. Moving a task from a cloud round trip (often one to two seconds end to end on a mobile network, as an illustrative figure) to local inference (hundreds of milliseconds or less) improves responsiveness and safety and also cuts the energy spent moving data across the network.

Industry verticals in 2025 and 2026

The effect of these advances is most visible in three sectors: automotive, industrial IoT and humanoid robotics.

Automotive: zonal architecture

Software-defined vehicles rely on high-performance SoCs such as NVIDIA DRIVE Thor and Qualcomm Snapdragon Ride. These enable zonal architectures, where a central computer handles perception and trajectory planning alongside cockpit functions, and zone controllers around the car handle local I/O.

Electric vehicle makers are using edge AI in battery management systems (BMS), where local models estimate battery health and manage charging in real time to extend battery life. Federated learning across fleets lets manufacturers improve driving models from real-world data without collecting raw driver data centrally.

Industrial IoT and manufacturing

Factories use edge AI to move from reactive to predictive maintenance. A sensor with a TinyML model can detect abnormal vibration on a motor and warn operators before it fails.

Vision at the edge allows real-time quality inspection on production lines. Compressed models on mid-range industrial SoCs keep frame rates high for defect detection without the cost or latency of a cloud connection.

Humanoid robotics

Humanoid robots are the next frontier. Platforms such as Jetson Thor are optimised for robot foundation models such as NVIDIA's Isaac GR00T, which need real-time fusion of lidar, cameras and microphones.

Running LLMs and VLMs on the robot lets it reason about its surroundings: understand a spoken instruction, find the right object in view and plan a grasp. Robot designers typically split the work so that the fast safety and motor control loops run in tens of milliseconds (a sub-50 ms budget is often cited for safe human-robot interaction), while the language and planning layers run more slowly above them.

Challenges and outlook

Several hard problems remain.

• The memory wall. The gap between compute speed and memory bandwidth is the main bottleneck for large models. Jetson Thor's 273 GB/s is a good example: for a model whose weights total 10 GB, generating each token means reading most of those weights, which caps output at roughly 27 tokens per second regardless of compute (an illustrative upper bound). Advanced packaging helps; TSMC has said it plans to keep expanding CoWoS capacity rapidly, with reports of a doubling by the end of 2026.
• Security. Risks range from model inversion and model theft to adversarial inputs. Secure enclaves, encrypted model storage and run-time checks are becoming standard requirements in enterprise-grade edge silicon.
• Software cost. Every new accelerator needs a compiler and runtime. Unified stacks are improving but are not yet plug and play.

The direction for 2026 and beyond is towards software-defined machines, where the line between device and cloud is set by cost, latency and privacy rather than by what the device can do. The combination of modular hardware (chiplets), unified compilers (MLIR) and decentralised learning (federated learning) is making intelligence a local property of devices rather than only a remote service.

The broader shift is from AI that assists to systems that act on their own at the point of action. The companies that best combine specialised semiconductors with adaptable software stacks are likely to lead the next cycle.

FAQ

What is the difference between edge AI and cloud AI?

Cloud AI runs models in data centres and sends results back over the network. Edge AI runs them on or near the device, which cuts latency and bandwidth and keeps data local, at the cost of tighter power and memory limits.

What is an NPU?

A neural processing unit is an accelerator designed for the matrix and tensor operations of neural networks. It runs them at much lower energy per operation than a CPU and is now standard in flagship phones and AI PCs.

Why is quantisation so important at the edge?

It shrinks models (four times from FP32 to INT8) and lets hardware use cheaper integer or low-precision arithmetic, which saves memory, bandwidth and energy, usually with little accuracy loss.

Where does RISC-V fit?

RISC-V's open, extensible instruction set lets companies add custom AI instructions and accelerators without licence restrictions, which suits custom edge and automotive silicon.