Training the model is the one-time cost. Answering your question is the bill that never stops

The usual way to talk about the cost of artificial intelligence is to quote a training run: the months of computation, the thousands of processors, the electricity consumed to produce one finished model. It is a genuinely large number and it is also, for any model people actually use, the smaller half of the story. Training happens once. Inference — the model answering a question — happens every time anyone types anything, forever, and it is the recurring cost that determines whether a service can exist at all.

Understanding why requires knowing what the hardware is really doing, and the answer is less exotic than the vocabulary suggests.

A neural network is mostly one operation

Strip away the terminology and a neural network is a long chain of matrix multiplications with a simple non-linear function applied between them. Multiply a block of numbers by a block of weights, bend the result slightly, repeat a hundred times. That is the forward pass, and training is the same thing run backwards to work out how each weight should change.

This shape is what makes the hardware question interesting. Matrix multiplication is millions of small multiply-and-add operations that are all independent of one another — no step waits for the result of another. You could do them in any order, or all at once.

A CPU is built for the opposite situation. It has a few very sophisticated cores designed to run a single sequence of instructions as fast as possible, with deep pipelines, branch prediction and large caches, because ordinary software is full of decisions that depend on the previous answer. Give it a million independent multiplications and most of that machinery is wasted.

A GPU makes the opposite trade: thousands of much simpler cores, all performing the same operation on different data at the same time. It is bad at branching logic and superb at doing one arithmetic pattern in enormous bulk — which happens to be exactly what a neural network is. Modern accelerators go further and build dedicated matrix-multiply units into the silicon, so that the fundamental operation of deep learning is a single hardware instruction rather than a loop.

There is a second lever: precision. Scientific computing traditionally uses 32 or 64 bits per number, but neural networks tolerate far less. Training now routinely runs in 16-bit or 8-bit formats, and models are served with weights compressed to 8 or even 4 bits. Halving the bits roughly doubles the arithmetic throughput and halves the data that must be moved — and moving data, it turns out, is the actual problem.

The limit is usually memory, not arithmetic

Here is the fact that reorganises most people's intuition: for a great deal of AI work, the processor is idle, waiting.

Arithmetic has become fast far more quickly than memory has become fast at delivering operands. A modern accelerator can perform arithmetic much more rapidly than its memory system can feed it, so performance is frequently set by memory bandwidth rather than by computing power. This is why accelerators are sold with stacks of high-bandwidth memory bonded onto the same package, and why the physical layout of a chip has become as important as its clock speed.

Text generation shows this at its starkest. A language model produces one token at a time, and each token requires reading essentially the entire set of model weights out of memory. For a single user's request, the chip performs relatively little arithmetic per byte it reads — the hardware spends most of its time streaming weights rather than multiplying.

The fix is batching. If you process many users' requests simultaneously, each set of weights read from memory is used for all of them at once. The same memory traffic now serves a hundred requests instead of one, and the cost per request collapses. This is why a large model served to millions of people can be economical while the same model running alone on a workstation feels wasteful — and why the engineering of a serving system is largely the art of keeping batches full without making anyone wait too long.

The interesting number is not how many operations a chip can perform. It is how many useful answers come out per unit of energy moved through memory — and that is a systems question, not a chip question.

Why per-query energy figures disagree so wildly

Published estimates of the energy used by a single AI query differ by two orders of magnitude, and most of that spread is not disagreement about measurement. It is that the question is underspecified.

The energy depends on which model answered — a small model tuned for a narrow task and a frontier model differ enormously. It depends on how long the response was, because generation cost scales with tokens produced. It depends on the input length, since attention over a long context costs more and the cached intermediate state grows with it. It depends on whether the request was batched with hundreds of others or served alone. It depends on the hardware generation and on how efficiently the data centre converts grid electricity into useful computing.

Quoting a single figure for "an AI query" is therefore about as meaningful as quoting a single figure for "a journey". The honest framing is that per-query energy is small in isolation and consequential in aggregate — which is precisely why the recurring cost dominates. A model trained once and then queried billions of times will, over its deployed life, consume considerably more energy in serving than it did in training.

That aggregate is now visible at the level of national infrastructure. Data-centre electricity demand has become a live planning constraint for grid operators in several countries, and siting decisions increasingly turn on power availability and cooling water rather than on land or connectivity. Rack power densities have climbed past what moving air can remove, which is pushing the industry towards direct liquid cooling — a change in the physical plumbing of computing, driven entirely by this workload.

What actually reduces the bill

Four levers matter, and none of them is "use a smaller model and accept worse answers".

Sparsity. Mixture-of-experts architectures hold a very large number of parameters but activate only a small fraction of them for any given token. Capacity grows without the per-token cost growing proportionally — the most consequential architectural efficiency idea of recent years.

Quantisation. Serving a model with weights stored at 8 or 4 bits instead of 16 cuts both memory footprint and memory traffic, which directly attacks the actual bottleneck. Done carefully, the quality loss is small; done carelessly, it is not, and knowing the difference is a real skill.

Distillation. Train a small model to imitate a large one on the specific distribution of tasks you care about. Most production systems do not need a frontier model; they need something that handles their traffic well, and a distilled model can be an order of magnitude cheaper to serve.

Moving work off the data centre. Phones and laptops now ship with neural processing units capable of running small models locally. Anything served on-device costs the operator nothing and never crosses a network.

Against all four sits an old pattern worth naming: efficiency gains tend to be spent on more usage rather than banked as savings. Every reduction in the cost per query has so far been followed by more queries, longer contexts and models asked to do more per request. Efficiency is necessary, but it should not be mistaken for a plan.

Why it matters for students and researchers

The centre of gravity in this field has moved. A decade ago the interesting question was the architecture; today, for almost everyone who is not at a frontier lab, the interesting question is serving — how to run a model on hardware you can afford, at a latency users tolerate, at a cost the service can sustain.

That is where the shortage of skills sits, and it is unusually relevant here. Very little frontier pretraining will happen in India; a great deal of fine-tuning, deployment and serving will, and those are the jobs. They reward people who understand memory hierarchies, batching, quantisation and profiling — systems engineering applied to machine learning, rather than model design.

It is also the layer where the sustainability argument becomes concrete rather than rhetorical. "Is AI bad for the environment" is not a question anyone can answer usefully. "How much energy does this deployment consume per useful answer, and what would batching, quantising or distilling it change" is a question with a number at the end of it, and getting to that number is ordinary engineering.

Frequently asked questions

Why does AI need GPUs instead of ordinary processors?

Because neural networks reduce to huge numbers of independent multiply-and-add operations. A CPU has a few complex cores optimised for sequential logic, while a GPU has thousands of simple cores that perform the same operation on different data simultaneously, which matches the workload exactly.

Does training or running an AI model use more energy?

Training is a large one-time cost, but for a widely used model the cumulative energy of answering queries exceeds it over the deployment lifetime, because inference happens continuously for as long as the service exists.

Why do estimates of the energy used by one AI query vary so much?

Because the figure depends on the model size, the length of the input and output, whether the request was batched with others, the hardware generation and the data centre's efficiency. Without specifying those, a single number for "one query" is not meaningful.

What is the memory bandwidth bottleneck?

Modern accelerators can perform arithmetic faster than memory can supply data. Generating text one token at a time requires reading the whole model repeatedly, so performance is limited by how fast weights move from memory rather than by computing power.

How can AI systems be made more efficient?

Chiefly by activating only part of a large model per request, storing weights at reduced precision, distilling a smaller task-specific model from a large one, batching requests to share memory traffic, and running small models directly on user devices.