The hidden cost of agentic AI is everything an autonomous agent needs to keep running: always-on inference in data centres, the electricity and water those centres consume, the advanced chips and the fragile supply chain that makes them, and the human workers who label, test and correct the models. Because agents loop through many model calls for every task, their running cost grows with use, not just with training. Autonomy for the user rests on a very large, mostly invisible base of silicon, power and people.
Key takeaways
- Agents run a continuous observe, reason, act loop, so inference, not training, becomes the main energy cost as use grows.
- One agent task can trigger tens of model calls, so energy per task can be many times that of a single chatbot answer.
- The hardware depends on a narrow supply chain: a few accelerator designers, EUV tools from ASML and leading-edge fabs mainly at TSMC.
- Data centre growth is straining local power and water in hubs such as Oregon, Singapore and Dublin.
- "Autonomous" systems still rely on human labellers, reviewers and safety teams, often low paid.
- Responsible autonomy means disclosing energy use, sourcing chips responsibly, paying human workers fairly and making agent behaviour traceable.
When ChatGPT first appeared, it felt as if intelligence had been packaged and handed out. A few years on, the next step is here: agentic AI, systems that plan, decide and act with little human input.
Agents do more than answer prompts. They pursue goals, work with other systems and sometimes negotiate on a user's behalf. The pitch is attractive: less friction, digital workers that never tire, and automation that scales.
But autonomy has a cost. Behind every agent sits a large stack of power-hungry computation, complex chips, human labour and infrastructure most users never see. This post looks at that side of the story: the data centres, the fabs and the people who keep the systems running.
How agents work
Earlier AI systems waited for a prompt and stopped when they had answered. Agents run continuously. They observe, reason, act and adjust, often without anyone watching each step.
That loop needs compute that is always on. An agent that manages your email, a portfolio or a logistics operation has to keep running inference, calling APIs and making decisions in real time.
Every one of those steps uses energy, and there are a great many of them across millions of users. Most discussion of AI's energy use has focused on training large models. With agents, the ongoing inference cost becomes the bigger concern. The models are not only large; they never stop running.
A worked example (illustrative)
Published estimates of the energy for a single chatbot text answer have fallen to well under one watt-hour. Google reported a median of about 0.24 Wh for a Gemini text prompt in August 2025, and Epoch AI estimated about 0.3 Wh for a typical GPT-4o query in early 2025. An agent is different, because one task can involve planning, tool calls, retries and long contexts. Using round, illustrative numbers:
| Step | Assumption (illustrative) | Result |
|---|---|---|
| Energy per model call | 0.3 Wh | 0.3 Wh |
| Model calls per agent task | 20 | 6 Wh per task |
| Tasks per user per day | 10 | 60 Wh per user per day |
| Active users | 10 million | 600 MWh per day |
| Average power | 600 MWh / 24 h | About 25 MW, around the clock |
That is roughly one large data centre running all day for a single, modest agent service, before cooling overheads, networking or the training runs behind the model. Change any assumption by a factor of ten and the answer moves by ten as well, which is why disclosure of real figures matters.
The compute behind each agent
A simple-looking conversation with an agent depends on data centres, GPUs and chip architectures built for massive parallel processing.
Each inference request draws power, and an agent that works continuously multiplies that many times over. If agents become as common as their backers expect, total compute demand could rise several times over in the next few years.
That power has to come from somewhere: from grids that are already stretched, and from chip supply chains that are increasingly shaped by geopolitics.
Silicon at the centre
To understand what agentic AI costs, start with the chips. They are the hardware that every agent ultimately runs on.
1. Specialised AI chips
For decades Moore's Law set the pace: smaller transistors, faster chips. AI changed the pattern. Instead of general-purpose CPUs, it depends on architectures built for tensor maths: GPUs, TPUs and other AI accelerators.
- NVIDIA leads with its H100 and Blackwell B200 chips, built for large-scale parallel workloads.
- AMD's MI300 and Intel's Gaudi 3 are competing hard on performance per watt.
- Start-ups such as Cerebras, Tenstorrent and SambaNova are trying new chip designs; Cerebras builds a wafer-scale processor that holds a whole model on one piece of silicon.
- Apple, Amazon and Google design their own silicon to control both efficiency and supply.
The common goal is not only speed but running AI efficiently at very large scale.
2. What it takes to make them
Making these chips combines extreme precision with global logistics. An advanced GPU holds tens of billions of transistors built on the most advanced process nodes.
The tools are just as demanding. ASML's extreme ultraviolet (EUV) lithography systems are among the most expensive machines in industry, with the newest High-NA models costing several hundred million dollars each, and their supply chain spans the Netherlands, Japan, Taiwan and the United States.
Each wafer from TSMC's fabs in Taiwan carries geopolitics as well as technology: US and China competition, export controls and the push to control advanced manufacturing. The race for agentic AI has become a race for the most advanced chips, and that race is changing the global order.
Where the cost sits in the stack
| Layer | Main cost | Who controls it |
|---|---|---|
| Accelerators | Design cost, scarce supply, high power per chip | NVIDIA, AMD, Intel, hyperscaler in-house chips, start-ups |
| Fabrication | EUV tools, leading-edge fabs, advanced packaging | TSMC, Samsung, ASML and its suppliers |
| Data centres | Electricity, cooling water, land, grid connections | Cloud providers and colocation operators |
| Networking | Low-latency switches and interconnect | NVIDIA (InfiniBand), Broadcom and others |
| Human work | Labelling, moderation, evaluation and safety tuning | Model developers and their contractors |
Strain on infrastructure
Building agents is not only about code. It is also about where the code runs.
Every agent depends on data centres, fibre networks and cooling systems. The same autonomy that users enjoy needs constant uptime, storage and inference.
1. Energy and water
A large data centre can draw tens of megawatts, comparable to a small town, and its cooling can use millions of litres of water a day. Agents, which run as millions of persistent processes, add to that pressure.
At a global level, the International Energy Agency's Energy and AI report (April 2025) estimated that data centres used about 415 TWh of electricity in 2024, around 1.5% of world demand, and projected that this could roughly double to about 945 TWh by 2030, with AI the main driver.
2. Concentration in a few places
Data centre hubs such as Oregon, Singapore and Dublin have faced pressure on local power and water as clusters have grown. Countries such as Saudi Arabia and the UAE are investing heavily in renewable-powered data campuses.
3. Networking and latency
Agents need fast, low-latency communication. That is driving demand for edge computing and AI networking hardware such as NVIDIA's InfiniBand and Broadcom's Tomahawk switches. The physical internet of cables, routers and switches is being rebuilt with AI traffic in mind.
The people behind "autonomy"
One irony of agentic AI is how much it still depends on people.
Behind systems sold as fully autonomous are workers who label data, moderate content, test edge cases and tune safety controls. Many are contractors in places such as Manila, Nairobi or São Paulo, often earning a few dollars an hour.
Human reviewers monitor, retrain and repair the models. Machines that "think for themselves" still rely on people who teach them to behave safely, and that work deserves to be counted as a real cost, not a footnote.
Who captures the value
Agentic AI changes how work gets done and who benefits from it.
1. A few platforms
Much of the power sits with a small group of companies:
- NVIDIA: the main supplier of AI compute.
- OpenAI, Anthropic, Google DeepMind and xAI: the developers of frontier models.
- Microsoft, Amazon and Google Cloud: the cloud providers that rent out AI capacity.
- TSMC and Samsung: the manufacturers of the most advanced chips.
Together they control not only the models but the silicon, the power and the cloud that agents run on.
2. Capital and carbon
Training a frontier model can cost hundreds of millions of dollars or more in compute and produce large carbon emissions. As agents multiply and run for longer, inference may overtake training in total energy use.
3. Compute as a national asset
Governments now treat compute as strategic. Examples include:
- the US CHIPS and Science Act, with about $52 billion for semiconductors;
- Europe's IPCEI programmes for microelectronics;
- China's domestic accelerator programmes;
- India's growing semiconductor and AI infrastructure.
The idea behind them is compute sovereignty: to control AI, a country needs control over the chips it runs on.
The complexity problem
Complexity is not only an engineering issue. It is a systemic risk.
Agents sit on many layers: hardware, firmware, model weights, orchestration frameworks and APIs. When they act on their own, it becomes hard to say who is responsible for a given outcome.
A fault, a biased dataset or unexpected behaviour can spread across many connected agents. Automated trading has already shown how quickly interacting algorithms can produce effects nobody intended. Managing that complexity now matters as much as adding capability.
What autonomy really means
We tend to think of autonomy as freedom from drudgery. In practice it also creates dependence: on infrastructure, on energy and on a fragile global supply of chips and data.
Real autonomy should include awareness of cost. Imagine AI systems that report not only accuracy and speed but also their energy, social and ethical footprint.
Towards responsible autonomy
The next phase needs transparency and sustainability at every layer:
- Energy accountability. AI services should disclose their operating energy use, as cars disclose fuel economy, and cloud providers should commit to renewable power and clear sustainability metrics.
- Responsible chip supply chains. Governments and companies should invest in ethical and sustainable sourcing, from mining to recycling. Circular design could become a real differentiator.
- Recognising human work. The people behind AI need fair pay and proper recognition. "Human in the loop" should mean partnership, not exploitation.
- Managing complexity. Autonomy should not mean opacity. Standards for tracing and explaining agent behaviour need to keep pace with the systems.
- Shared responsibility. Companies, developers and users all share accountability for what agents do. The more we delegate, the more we need to understand.
FAQ
What is agentic AI?
AI systems that pursue goals by planning and taking actions, such as calling tools, browsing or sending messages, rather than only answering a single prompt.
Why does agentic AI use more energy than a chatbot?
Each task can involve many model calls, longer contexts and continuous monitoring, so the inference energy per task is much higher than for one answer.
Does inference or training use more energy?
Training a frontier model is a large one-off cost; inference is paid every time the model is used. As agents spread and run continuously, total inference energy can overtake training.
How can the cost of agentic AI be reduced?
More efficient chips and models, smaller models for routine steps, caching, renewable-powered data centres, and clear reporting of energy use so buyers can compare services.
Conclusion: the price of thought
Agentic AI turns software from a tool that waits for instructions into one that acts on its own. That shift rests on a large and mostly invisible base of infrastructure.
The chips, data centres, human labour and natural resources behind it form a new economy, measured not only in dollars but in watts, water and trust.
Every act of intelligence consumes something. As we build systems that think for us, we should make sure they also work with us: openly, responsibly and sustainably.
Autonomy is not free. It can be fair, if we look at the whole system that powers it.
Well written article Murugavel. It covers major aspects of Agentic AI and the industry advances with a crisp and informative outlook. Thank you for this post.
ReplyDelete