What an Inference-First Compute Center Looks Like on a Power Meter

Two AI compute centers can carry the same megawatt nameplate, the same GPU count and the same cooling design, and still look nothing alike on a power meter. One holds a steady, high block of load for weeks and then drops by a third in a second. The other breathes: it flickers between idle and near-peak thousands of times per hour, follows the sun across time zones, and empties out every night. The first is a training hall. The second is an inference hall — and as the industry's center of gravity shifts from training runs to serving tokens, the difference between those two signatures has become a facility design problem, not a curiosity.

This article works through the energy characteristics of servers in inference-oriented and large-model compute centers: what the silicon actually draws in each workload, why the two phases of a single inference request behave like different machines, how reasoning-style outputs multiply the bill, and what each of these signatures demands from power delivery, cooling and capacity planning. The arithmetic is all reproducible from published measurements, and the references are linked at the end.

Start at the silicon: static, dynamic, and a ceiling

Every GPU power conversation starts with a decomposition, because three different numbers hide behind one datasheet. The static part is leakage and the always-on board infrastructure — memory refresh, management controllers, clocks. It is paid whenever the device is powered, whether or not it does useful work. The dynamic part is switching energy, which grows steeply with clock frequency. And the thermal design power is a ceiling, not an operating point.

The static fraction is larger than most people expect. An H100 SXM idles at roughly 100 W against a 700 W rating — about one-seventh of nameplate, consumed by an apparently "off" chip [1][2]. At the server level the picture compounds: fans alone can account for roughly a quarter of server power in high-density systems, and a rated 6.5 kW DGX-class host in a large cloud study never actually exceeded 5.7 kW across its workloads — 800 W of provisioned capacity per server that was real copper feeding nothing [3].

Two consequences follow, and they drive everything else in this article. First, a powered GPU waiting for work is not a free zero: an inference fleet at low traffic carries a large fixed electrical floor, and every unserved hour spreads that floor across fewer answers. Second, because the static term is paid per unit time rather than per unit work, the energy cost of one token falls almost in proportion to how many tokens share the device at once — which is why batching, context length and scheduling dominate the per-token energy story long before chip generation does [4].

There is also a measurement subtlety worth naming: researchers studying production clusters distinguish between idle time and execution-idle — time when a GPU shows activity but little useful progress. Across studied clusters that state covered about 19.7% of in-execution time while contributing 10.7% of energy: real power spent at low utilization, invisible to any dashboard that only tracks "is a kernel running" [5].

The training signature: a block load with a sawtooth on top

Training a large model is not a steady load either, but its variation has a fixed anatomy. The job runs for days to weeks at high utilization, drawing near the power ceiling. At intervals it hits a checkpoint: parameters are written to storage, compute pauses, and power drops sharply — but not to zero. GPUs hold a high floor, commonly 60–70% of peak, so the job can resume instantly. Then power ramps straight back to the plateau. The result is a sawtooth around a high mean, repeating for the life of the run [6].

At cluster scale the swings get serious. A characterization study of production LLM fleets at a major cloud provider measured training power swings of up to 37.5% of provisioned capacity within two seconds, as synchronized jobs moved between compute phases and gradient synchronization across thousands of accelerators [3]. To a power delivery system, that is a recurring step change the size of a small substation load, deliberately repeated.

The energy totals are public for one frontier model, which makes the arithmetic worth doing once. Meta's Llama 3.1 405B consumed 30.84 million H100 GPU-hours at a 700 W rating — the GPU-side energy alone is:

The whole Llama 3.1 family took 39.3 million GPU-hours — 27.5 GWh of GPU-only energy before facility overhead [7]. Those numbers are worth internalizing at procurement time: a single frontier pretraining run consumes the continuous output of roughly nineteen 132 kW racks for a year, spent once, to produce one artifact.

The inference signature: two clocks, neither of them training's

Inference load is shaped by two clocks at once, and neither looks like a training profile.

Clock one is human traffic. An interactive serving fleet follows a diurnal envelope: request rates rise through the day, peak in the evening, and fall into a deep overnight trough. The same cloud fleet study found inference power changes by at most ~9% of provisioned capacity within two seconds — an order of magnitude calmer than training on that timescale — while the meaningful variation happens over hours [3].

Clock two is the request itself. Each incoming request drives the GPU from idle to high power and back in fractions of a second. Individual accelerators cycle between roughly one-seventh of nameplate and near-peak thousands of times per hour; at the chip level that is an idle-to-peak swing of three to seven times depending on generation and workload [2][6][8]. The saving grace is decorrelation: because requests arrive on independent schedules across thousands of serving endpoints, their peaks statistically multiplex rather than align, which is why a fleet's aggregate draw is much smoother than any single GPU's — and why inference clusters, unlike training clusters, have genuine headroom for power oversubscription [3].

Inside one request: prefill and decode are different machines

The single most consequential fact about LLM inference energy is that one request contains two phases with opposite bottlenecks, and they run on the same silicon anyway.

Prefill processes the entire input prompt as a large parallel batch of matrix math. It is compute-bound: it saturates the tensor cores and uses the GPU's power budget efficiently, running at around 90% utilization [9][10]. Decode generates output tokens one at a time, each step dominated by reading the key–value cache from memory rather than by arithmetic. It is memory-bound: even under state-of-the-art batching, decode typically reaches only 15–35% utilization, and its power draw sits flat and stable well below the ceiling — decode rarely approaches TDP because it cannot keep the arithmetic units busy enough to burn the power the clock is feeding them [4][9][11].

The Splitwise study at Microsoft quantified how wasteful the mismatch is: capping the power of decode machines by more than half barely moved latency, because decode was not using the compute that power was paying for. Splitting prefill and decode onto separately provisioned machine pools delivered up to 2.35× the throughput within the same power budget [9][10]. The hardware angle is the same story at generation scale: from A100 to H100, compute grew 3.43× and power 1.75×, but HBM bandwidth only 1.6× — every generation widens the gap that decode cannot exploit [9].

The reasoning tax: output length is the energy dial

Per-query energy for a frontier-scale chatbot exchange is small and now well-measured. The careful numbers cluster around one number: a Microsoft Research bottom-up study in Joule put the median at 0.31 Wh per typical chatbot query (interquartile range 0.16–0.60), consistent with OpenAI's self-disclosed 0.34 Wh, Epoch AI's first-principles estimate of ~0.3 Wh, and Google's comprehensive 0.24 Wh figure for a median Gemini prompt [5][12]. The energy per answer is not the interesting part. What happens to it is.

Reasoning-style outputs multiply it by an order of magnitude or more. Long chains of thought generate ten to fifty times more output tokens per response, and longer sequences consume KV-cache memory that prevents large batches, raising the energy per token as well. The Joule study found reasoning queries with ~5,000 output tokens raise energy per query roughly 13× to 3.91 Wh. The ML.ENERGY benchmark measured it directly across 46 models on H100 and B200 hardware: problem-solving (reasoning on) averaged 25× the energy per response of plain chat — 6,988 output tokens at 4,625 J versus 717 tokens at 184 J [13][14]. On the extreme commercial end, long-prompt measurements of DeepSeek-R1 and o3 exceed 33 Wh per query — seventy times an efficient small model [12].

Table 1 — Energy per query is a function of boundary, task and model. Any single "AI energy per query" number without these three qualifiers is noise [5][12][13].

The fleet-level consequence is the number facility planners should keep. Serving one billion standard queries per day costs roughly 0.7 GWh; if 10% of traffic is long reasoning queries, demand rises to 1.7 GWh per day — more than double from one-tenth of the requests [12]. A hall whose traffic mix shifts from chat to agents and test-time scaling does not need more racks to double its energy bill; it needs only its users to change behavior. Conversely, the measured mitigation levers are large: B200-class hardware cuts median energy ~35% versus H100 at matched latency, and model/serving efficiency pathways point to 8–20× reductions per query [12][5].

Context is the other multiplier: the 1/W law

Between the chip and the application sits a third determinant of inference energy that has nothing to do with hardware generation: how long a context window the service offers. Doubling the context window halves the tokens a fleet delivers per watt. The mechanism is memory, not arithmetic: a larger per-request KV footprint leaves less room for concurrent sequences, so batch size falls while the GPU's static draw stays on [15].

The measured spread is dramatic. On identical H100 hardware, serving at 4K context fits 256 concurrent sequences at 17.6 tokens per watt; at 64K context the same GPUs fit 16 sequences at 1.5 tokens per watt — an 11.7× efficiency gap between two configurations of the same fleet (the underlying study reports up to 40× across deployment variants) [15]. Routing matters more than silicon: a two-pool strategy that segregates long-context and short-context traffic recovers ~2.5× tokens-per-watt, while upgrading the whole fleet from H100 to B200 is worth ~1.7×. Mixture-of-experts models compound it further — Qwen3-235B-A22B activates 22B parameters per token and reaches 37.8 tokens per watt at 8K context, 5.1× a dense 70B [15].

The procurement translation is uncomfortable but useful: system-level routing and context policy can beat a hardware refresh, and an inference hall's energy intensity is set by its traffic contract as much as by its GPUs. A colocation buyer whose tenant serves 128K-context agents is buying a different building than one whose tenant serves 4K-context search snippets, even at identical rack counts.

Translating load shape into steel: the facility deltas

Everything above compresses into a short list of consequences that change what gets built. The reference rack makes the stakes concrete: a GB200 NVL72 draws 125–135 kW continuous through direct-to-chip liquid cooling with zero rack fans, against an electrical design peak of up to 192 kW — HPE's guidance is to provision the busway to the peak, roughly 1.45× the nominal draw [16][17]. Of the ~132 kW nominal, about 115 kW lands on the facility water loop and ~17 kW remains in the air path: even "fully liquid-cooled" racks leave 13% of their heat for the room [17].

Table 2 — The same hardware serves both, but the halls that house it are engineered differently. Sources as cited.

Four of these rows deserve expansion, because they are where money moves.

Cooling sees the past, not the present

A heat exchanger has thermal mass and a control loop with dead time. When training power drops at a checkpoint, the cooling plant keeps working at the pre-drop level for minutes; when an inference fleet's evening peak passes, chilled-water demand lags into the night. Facility-level demand curves are therefore smoother than IT curves in both directions — which flatters nobody's PUE measurement at part load. The part-load penalty is not cosmetic: pumps riding their curves and chillers cycling below their sweet spot can push facility overhead up sharply in an empty hall, a dynamic we quantify in our piece on long-term low IT load in liquid-cooling systems.

The UPS sees two different stress patterns

Training presents the UPS with a steady block load and rare, large step changes; sizing is a thermal and duration problem. Inference presents constant millisecond-scale power ripple on top of a diurnal swing — statistically gentler at the bus, but with far more charge/discharge micro-cycling if the topology lets the battery follow load. Neither profile resembles classic enterprise IT, and both belong in the sizing conversation before equipment is ordered, not after (see our UPS battery sizing walkthrough for AI halls).

Provision to the transient, not the nameplate

The 192 kW busway figure for a "132 kW" rack is not conservatism; it is the measured reality that synchronized GPU workloads overshoot their own average by half again [16][17]. The same logic scales up: a hall of training racks needs switchgear and transformer margins built for 37.5%/2-second steps, while an inference hall's margins buy statistical multiplexing instead — the same copper, spent differently. Power delivery architecture is converging on this problem (our 800 VDC architecture piece covers the distribution-side arithmetic).

The overnight trough is an asset with a scheduling policy attached

An inference hall's night trough is where evaluation runs, batch inference, fine-tuning and retraining belong. Filling it converts a liability — static power spread over few tokens — into the cheapest compute the facility will ever sell. Operators who treat the trough as spare capacity rather than idle margin recover a meaningful slice of the 60–70% average utilization that production fleets otherwise leave on the table [3][5].

A buyer's checklist for inference-first compute

• Demand the load profile, not the nameplate. Ask tenants for expected mean-to-peak ratio, 2-second swing envelope, and diurnal curve. A 1 MW lease that peaks at 0.55 MW and a 1 MW lease that runs flat are different buildings.
• Ask which phase the hall serves. Prefill lanes tolerate dense liquid-cooled packing; decode lanes want memory-dense, power-cappable hardware and reward aggressive oversubscription. If the provider cannot answer, they have not characterized their own load.
• Pin the traffic mix in the deal. A 10% shift from chat to reasoning-style traffic can double facility energy [12]. Index energy pass-throughs or expansion triggers to token mix, not just rack count.
• Size cooling for capture fraction and lag. On a fully liquid rack, ~13% of heat still enters the room air [17]; the water loop carries the rest with a control lag measured in minutes. Both facts belong in the plant model.
• Require power capping and telemetry as a delivered feature. The derating studies found 800 W/server of phantom provisioned power and fans worth a quarter of server draw — none of it recoverable without out-of-band power control [3].
• Treat context policy as an energy contract item. An 11.7× tokens-per-watt spread sits between 4K and 64K context on identical silicon [15]. Who controls routing controls the power bill.

Bottom line

The energy character of an AI compute center is set by workload phase before it is set by chip count. Training halls are block loads with a sawtooth: enormous, one-time energy concentrated into a sustained high floor — a single frontier pretraining run is ~26 GWh at a modern PUE, nineteen rack-years in a day. Inference halls are living systems: a static floor that never sleeps, sub-second flicker that multiplexes into calm, a diurnal envelope that follows people, and an energy bill dominated by output tokens and context length — where one-tenth of traffic can double consumption and a routing decision can beat a hardware generation.

The industry's electricity story is increasingly the second hall, not the first: data centers consumed about 415 TWh in 2024, on a path toward roughly 945 TWh by 2030, with AI-focused facilities growing fastest — and the lifecycle energy of a deployed model is dominated by inference within months of its training run ending [5][12][18]. Buyers who learn to read the three load shapes in Figure 2 — and to buy steel accordingly — will price AI capacity more accurately than the market around them.

Planning a modular hall for inference or training duty? Tell us the load shape you need to house — we work from the meter curve backward.

References

1. NVIDIA, "H100 Tensor Core GPU" product page — SXM5 700 W rating. nvidia.com
2. Massed Compute, "H100 SXM5 Power Consumption Breakdown by Mode" — idle 50–100 W, inference 300–500 W, training 600–700 W. massedcompute.com
3. Stojkovic et al., "Characterizing Power Management Opportunities for LLMs in the Cloud" (ASPLOS'24) — 37.5% vs 9% two-second swings, diurnal inference pattern, 6,500 W rated vs 5,700 W observed, fans ~25% of server power, inference oversubscription headroom. foci.uw.edu (PDF)
4. The Machine Learning Society, "Energy-Optimal Inference Serving: A Cost Model" — static/dynamic power decomposition, idle ~1/7–1/8 of TDP, decode never near the ceiling, static energy per token falls with batch. tmls.nyc
5. IntuitionLabs, "Energy Use per AI Inference Task: Joules and Watt-Hours" (2026) — Google 0.24/0.10 Wh boundaries, OpenAI 0.34 Wh, Epoch AI ~0.3 Wh, IEA 415→945 TWh, eight swing variables, B200 −35% vs H100. intuitionlabs.ai
6. Information Technology and Innovation Foundation, "Five Concerns About AI Data Centers" (2026) — training sawtooth with 60–70% checkpoint floor, University of Alberta GPT-2/Llama 3.1 measurements, transformer thermal fatigue, inference flicker and voltage-sag risk. itif.org
7. Meta, Llama 3.1 model cards — 39.3M H100 GPU-hours family total; 30.84M for 405B at 700 W TDP; 8,930 tCO2eq location-based. huggingface.co
8. OPAL-RT, "AI Workload Variability and Its Impact on Data Center Power Stability" — per-workload profile table (pretraining/finetuning/inference), synchronized ramp events, cooling response lag. opal-rt.com
9. Patel et al. (Microsoft/UW), "Splitwise: Efficient Generative LLM Inference Using Phase Splitting" (ISCA 2024) — prefill compute-bound vs decode memory-bound, >50% decode power cap with negligible latency impact, A100→H100 3.43× compute vs 1.6× HBM bandwidth. arxiv.org (PDF)
10. Choukse et al., "Splitwise: Phase-Split LLM Inference" (IEEE Micro, 2025) — up to 2.35× throughput at the same power and cost budget. computer.org
11. "Phase-Decoupled, Model-Calibrated Power Control for Disaggregated LLM Serving" (arXiv, 2026) — decode lanes draw flat, stable power well below TDP; phase disaggregation makes the phase distinction spatial; 9–12% datacenter power savings from phase-aware control. arxiv.org
12. Stojkovic et al., "Energy Use of AI Inference: Efficiency Pathways and Test-Time Scaling" (Joule, 2026), figures via getshim.tech synthesis and IntuitionLabs [5] — median 0.31 Wh/query (IQR 0.16–0.60), reasoning 3.91 Wh (~13×), 1B queries/day 0.7→1.7 GWh with 10% long queries, 8–20× reduction pathways; URI 30-model measurements (DeepSeek-R1/o3 >33 Wh long-prompt).
13. ML.ENERGY, "Diagnosing Inference Energy Consumption with the ML.ENERGY Leaderboard v3.0" (Dec 2025) — 46 models / 7 tasks on H100+B200; reasoning-on 6,988 output tokens & 4,625 J/response vs 717 & 184 J (25×); KV-cache memory limits batch size; energy/token 0.151–0.312 J. ml.energy
14. AI Tech Model, "How Does Idle GPU Capacity Affect the Electricity Cost of AI Answers?" — execution-idle 19.7% of in-execution time / 10.7% of energy (CMU-Princeton 2026 preprint); Uptime 1.56 fleet-average PUE. aitechmodel.com
15. Chen et al., context-window energy study, covered in "Energy Efficiency in LLMs Hinges on Context Window Size, Not Just Hardware" (Energy Innovation Review, 2026) — 1/W law; H100 4K ctx 256 seq / 17.6 tok&W vs 64K ctx 16 seq / 1.5 tok&W; FleetOpt 2.5×; B200 ~1.7×; Qwen3-235B-A22B 37.8 tok&W (5.1×). energyinnovationreview.com
16. GPU Smith, "NVIDIA GB200 NVL72" verified datasheet — 125–135 kW continuous, peak/transient up to 192 kW (HPE EDPp, ~1.5× TDP), 100% direct-to-chip, zero rack fans, ASHRAE W-series supply 18–25 °C. gpusmith.com
17. SecondWatt, "GB200 NVL72" — 132 kW nominal, 192 kW busway provisioning, heat split 115 kW liquid / 17 kW air, ~1.42 MW of busway per IT MW. secondwatt.com
18. International Energy Agency, "Energy and AI" and 2026 follow-up — data centres ~415 TWh (2024), ~945 TWh by 2030; AI-focused DC electricity +50% in 2025, projected to triple by 2030; figures via IntuitionLabs [5] and getshim.tech.