From data to leaks: How AI is eating up the world’s power

March 4, 2026

blog

From data to leaks: How AI is eating up the world's power

American data centers were able to keep their electricity usage surprisingly stable from 2005 to 2017, but constant improvements in electronics kept annual increases to a fairly small amount. Then, around 2017, AI arrived in force and disrupted that stability.

AI required another computing machine designed for huge mathematical workloads, not for regular tasks like our beloved PCs. Data centers began to fill up with specialized processors called GPUs. GPUs were originally built for video game graphics, but they’re perfect for computing the massive amounts of math needed to train AI models. These processors are incredibly powerful, but they’re also energy hungry.

As a result of this change, by 2023, U.S. data centers will use twice as much electricity as they did just 10 years ago. Analysts from the IEA and Gartner suggest that by 2030, data centers could consume as much electricity as Japan or Germany.

Electricity usage in U.S. data centers remained stable from 2005 to 2017, with small increases each year. The stability has been broken since 2017. (Source: 2024 U.S. Data Center Energy Use Report, Berkeley Lab).

This raises a natural question. Why does AI require so much energy? What’s going on inside these machines that consumes so much power?

The energy cost of learning: Why AI training consumes so much power

Imagine how a traditional computer handles tasks. Similar to when a cashier scans an item at a register, it goes through a step-by-step sequence of instructions: one barcode, one beep, then the next barcode. Workloads are orderly, predictable, and relatively contained.

Training modern AI models is nothing like that. Instead, imagine trying to scan every item in an entire supermarket at the same time, not once, but many times, thousands or even millions of times. Now imagine that a “supermarket” contains billions of data points (words, images, audio clips, video frames), and each step through this vast inventory requires trillions of mathematical calculations. This is what AI training at scale looks like.

Large-scale AI models are built on deep neural networks with billions or even trillions of parameters. During training, the system iteratively adjusts these parameters through a process called backpropagation, comparing predictions to ground truth and incrementally fine-tuning them. Each adjustment requires a huge number of matrix multiplications, which are computationally intensive and need to be performed in parallel on thousands of dedicated processors.

And this process runs for months in huge data centers packed with high-performance AI accelerators. These chips continuously consume megawatts of power. The dense concentration of processors generates a tremendous amount of heat, which requires more energy to cool the hardware.

Always on, always drawing power: The energy reality of AI inference

Although training uses a lot of energy, it is not the greatest continuous power demand. Training is an intensive surge, a large burst of computation over weeks or months. In contrast, inference is continuous. It really doesn’t stop.

Inference is what happens after the model is trained. Every time someone asks an AI system a question, the model performs inference, the act of generating a prediction or response from what it has already learned.

A single query takes only a fraction of a second. On a global scale, these fractions add up. That one interaction is multiplied by billions of queries per day. The demands become relentless.

Unlike training, which ends eventually, inference works permanently. AI systems reside in data centers around the world, waiting for your next request. And next. They need to respond immediately. This means that the infrastructure that powers them must remain active and ready at all times.

Industry analysts estimate that approximately 80-90% of today’s AI-related computing is spent on inference rather than training. The result is a continuous high baseline energy demand.

Energy breakdown: What actually consumes power in AI systems

Surprisingly, the biggest energy consumer is not the actual mathematical thinking that AI does. It’s just moving data around that consumes a lot of energy. If computer chips are cities, computations are the silent conversations people have with each other, and data movement is the traffic on highways, roads, and alleys. Conversations use very little energy. On the other hand, traffic jams consume fuel. Inside the AI processor, the imbalance is just as dramatic. Moving data requires thousands of times more energy than performing the computation itself.

Professor Mark Horowitz of Stanford University quantified this several years ago. He showed that performing small-scale calculations on a chip consumes as much energy as lighting a small LED for a fraction of a second. Sending and receiving data between a chip’s memory and processor is like turning on an old incandescent streetlight. This comparison is not even close. Imagine repeating that a trillion times. The costs can add up surprisingly quickly.

Energy dissipation between components varies widely, from less than 1 picojoule for integer operations in adders and multipliers to several picojoules for floating-point operations. Memory access turns out to be even more expensive. Retrieving data from cache consumes 20 to 100 picojoules, while accessing DRAM consumes up to three orders of magnitude more, exceeding 1,000 picojoules (Source: Stanford University).

Throughput inefficiency: When the AI chip waits without computing

While some of the energy imbalance in AI systems is rooted in semiconductor physics, where moving data consumes more energy than performing computations, another key factor is the uneven pace of technological advancement.

Over the past 30 years, processors have advanced at an incredible rate. The number of transistors per chip has skyrocketed, parallelism has exploded, and dedicated AI accelerators can now perform an astonishing number of operations per second. Computing power has increased dramatically.

Your memory system just can’t keep up. This widening gap is known as the “memory wall.” While processor speeds have improved dramatically, improvements in memory bandwidth (how fast data can be delivered) and memory latency (how fast data can be accessed) have been much slower. Two curves diverge. And that disconnect is at the heart of AI throughput inefficiencies.

The illusion of peak performance

In theory, modern AI processors could advertise performance in the hundreds of teraflops or even petaflops. Theoretically, it can perform 1 quadrillion operations per second.

In reality, because the processor spends so much time waiting for data to arrive, actual performance is often only a fraction of that, sometimes less than 10-20% of peak capacity when running large AI models.

Imagine owning a sports car that can go 320 mph, but you’re stuck in traffic almost all the time. The engine is powerful. The design is sophisticated. But the road infrastructure cannot support that speed. “Traffic” in AI systems is the movement of data.

Parameters must be fetched from memory. Activation must be written back. Intermediate results must be transferred across chip boundaries. Each of these steps consumes time and energy. If the data cannot be delivered fast enough, the computing unit shuts down and continues consuming power without doing any useful work. Unused computing power doesn’t just disappear quietly. That’s thermal overhead.

architecture mismatch

The root cause lies in computing architecture.

Pervasive CPUs are designed for sequential general-purpose tasks. They are great for control logic, branching, and diverse workloads, but are bad at feeding thousands of parallel compute units simultaneously.

Conversely, the flagship AI processing GPUs were originally designed for image rendering, processing large amounts of data in parallel. They are much better suited for AI workloads than CPUs, but even GPUs inherit memory hierarchy and data movement assumptions formed by graphics applications, not neural networks.

Modern AI workloads aren’t just “bigger.” They are structurally different.

Large-scale neural networks require large matrix multiplications, repeated reuse of model parameters, very high memory bandwidth, tight synchronization between parallel units, and rapid movement of data between chips and racks. The demand for data delivery is growing faster than the industry’s ability to efficiently deliver it.

As a result, today’s AI accelerators resemble elite athletes with extraordinary muscle mass, but insufficient blood flow. Their computational engines are huge, but their circular systems, the paths that move data between memory and computation, can’t keep up.

Data movement: hidden energy costs

In many AI workloads, moving data consumes more energy than performing the computation itself.

Fetching a value from off-chip memory can require orders of magnitude more energy than multiplying two numbers. When scaled over billions or trillions of operations, this imbalance dominates the overall power consumption. This is why throughput inefficiency is not just a performance issue, but an energy issue.

Until memory systems, interconnects, and computing architectures evolve in tandem, AI processors will continue to operate far below their theoretical limits, consuming large amounts of power not only in computation but also in standby.

Conclusion: Rethinking AI hardware for a sustainable future

Solving this problem requires more than incremental improvements to existing computing architectures. Gone are the days when performance improvements could be achieved simply by adding more compute units. AI has revealed deeper architectural constraints. Computing is no longer the primary cost. Data movement.

The next generation of AI processors will not be defined by their ability to theoretically perform trillions of operations per second. Instead, they are measured by how efficiently they coordinate data flow, minimize movement, localize memory accesses, and intelligently reduce idle cycles.

Just as modern cities redesign their infrastructure around traffic flow, the semiconductor industry needs to start designing chips around data flow. Adding lanes to freeways will not solve congestion if bottlenecks remain at freeway entrances and exits. Similarly, adding more compute units will not solve AI efficiency problems if memory bandwidth and interconnect architecture remain constrained.

Today’s large-scale AI systems often operate at approximately 10-20% of their advertised peak performance under real-world workloads. Raising this number to 80-90% would not only represent a quantum leap in performance, it would also represent an energy revolution, dramatically reducing wasted power, lowering cooling requirements, and enabling sustainable expansion.

This change requires not only manufacturing improvements but also architectural innovations. Calculations abound. Moving data is expensive. The sooner hardware architects recognize that truth, the more sustainable and scalable the future of artificial intelligence will be.

Lauro Rizzatti is a business development executive at VSORA, a pioneering technology company delivering silicon semiconductor solutions that redefine performance, and is a renowned chip design verification consultant and industry expert in hardware emulation.