GPU Server Burn-In Testing: What to Run Before Racking

A GPU server is the most expensive part per rack unit that most data centers will ever install, and it arrives already carrying the failures it will have for the rest of its life. Electronics follow a bathtub curve: a high failure rate in the first days of operation, a long flat middle, and a rising tail at end of life. The early region is not theoretical — it is where marginal solder joints, under-torqued cold plate mounts, weak voltage regulator modules, and mis-seated connectors declare themselves, and it is the only part of the curve a buyer can compress on purpose [8][10].

Burn-in testing is the deliberate compression. This guide covers what to run on a GPU server before it is handed production work: the sequence, the tooling, the diagnostic levels worth running, the error counters that mean "replace this GPU" versus "watch this GPU", the NCCL bandwidth numbers a healthy fabric should produce, the exit criteria that define a pass, and what the whole exercise costs in time and electricity.

What "burn-in before racking" actually means

The phrase covers several distinct gates, and conflating them is how a deployment ends up with a proof of nothing. In order from the factory outward:

The pre-ship stages are complementary, not alternatives: burn-in catches electrical and thermal faults, firmware and I/O verification catches configuration and link-training errors, automated optical inspection catches bent pins and mis-cabled ports, and in-transit monitoring catches shock and humidity excursions. Skipping any one accepts that class of failure as a field risk — rarely a good trade on a rack that draws anywhere from 30 to 100+ kW [9][14].

This is also a different exercise from facility load bank testing. Load banks prove the power and cooling infrastructure with a resistor standing in for IT load; burn-in proves the IT hardware with real compute standing in for production load. The two run at different times, against different acceptance criteria, and a site generally needs both. Our step-by-step load bank testing procedure covers the facility side.

Why 48 to 72 hours — and why four hours is not a test

The purpose of duration is thermal soak and thermal cycling. Solder and connector faults, marginal fan bearings, and coolant fitting weeps typically need hours at temperature and several heat-up/cool-down cycles to manifest. A four-hour pass proves the hardware powers on and computes; it does not reproduce the conditions that make a weak part fail [6][8][10].

Working practice across operators clusters around 72 hours minimum, 5–7 days for large or liquid-cooled deployments, with at least two deliberate power cycles somewhere in the campaign — a cold restart finds a different fault class than steady-state operation, and the last 48 hours must be error-free to count. If a fix is applied on day four, the clock restarts for the affected node [6].

The payoff is measurable: a 48–72 hour burn-in window typically identifies 90%+ of latent manufacturing defects before the node enters the production pool [8]. The cost of the alternative is a weak GPU discovered three weeks into a month-long training run.

One practical consequence: burn-in takes power, and it takes it continuously. A modern liquid-cooled AI rack at 132 kW held for 72 hours consumes roughly 9,500 kWh at the rack, about 12,400 kWh with a 1.3 PUE — on the order of $1,500 in electricity at $0.12/kWh. Scale that to an eight-rack pod and the campaign burns about 98,800 kWh, roughly $11,900 — the energy equivalent of nearly 1,900 gallons of diesel. This is not wasted power, but it has to be planned into the commissioning schedule, the facility power budget, and the CDU or cooling plant's load shape [9].

Step 1 — Bring the rack up in the right order

The physical sequence matters more than it looks. Power servers sequentially, not simultaneously — bringing a full rack online at once creates an inrush current that can trip upstream breakers. Spacing power-on by 30 to 60 seconds per server lets the electrical infrastructure absorb the load gradually [14].

Before any workload runs, the configuration baseline gets set and recorded:

• Boot order — network PXE for automated provisioning, or local disk, decided deliberately rather than defaulted.
• Memory speed and ECC settings — confirm the modules trained at rated speed.
• PCIe slot configuration — confirm every GPU and NIC enumerates in the intended slot, at the intended generation and width.
• Fan and power capping profiles — if the site operates within a per-server power budget, cap deliberately and record the setting.
• Firmware matrix — BMC/IPMI, BIOS/UEFI, GPU vBIOS, NIC (ConnectX for InfiniBand or Ethernet), and NVMe drive firmware, all updated to the validated versions in the correct order, then verified. Some firmware updates must be sequenced to avoid compatibility issues [14].

Verify component recognition before testing anything: GPUs visible with correct memory and compute capability through the driver, HCAs reporting expected rates, every drive enumerated. Record serials, firmware versions, and part numbers as part of the asset record — that inventory is the reference every later comparison depends on [5][14].

Step 2 — The smoke test: DCGM diagnostics, levels 1 through 4

NVIDIA's Data Center GPU Manager (DCGM) ships the canonical active health diagnostic, dcgmi diag, with four suite levels of increasing depth and duration [1][3]:

The commissioning standard that has emerged in practice: level 3 on every GPU as an acceptance gate, level 4 at least once during the burn-in campaign, and level 1 as a recurring pre-flight check. Archive the full JSON output per GPU — not a screenshot of "PASS" — because the JSON is the baseline that later drift gets measured against [1][5].

For an eight-GPU node, the levels run against all GPUs in parallel, so wall-clock time stays close to the per-GPU figure: about 15 minutes for level 3, about 90 minutes for level 4. Running them strictly one GPU at a time on a 64-GPU pod would take roughly 16 hours for level 3 alone — a reminder to parallelize the diagnostic sweep and keep the serialization for cross-node fabric tests [1].

Step 3 — Thermal soak under sustained load

After the smoke test, the node runs a compute load near its thermal design power for 24 to 48 hours with all nodes loaded concurrently, watching for three things: thermal throttling, power supply trips, and memory errors [6][8].

The thermal part is not a formality. Sustained maximum-load testing is how you find inadequate thermal interface material application, vapor chamber defects, or cold plate mounting inconsistencies in a direct-to-chip loop — all of which pass a functional test and show up as performance loss later [8]. What you are looking for is junction temperature and hot-spot behaviour at the design ambient, not a single temperature reading.

A 50 kW rack under burn-in rejects on the order of 170,000 BTU per hour — matching the figure integrators publish for AI burn-in cells, and roughly an order of magnitude above the 5–10 kW per rack that precision air cooling was originally designed around. The burn-in cell therefore needs in-row cooling, rear-door heat exchangers, or a direct-to-chip liquid loop to sustain a realistic workload at all [9].

For liquid-cooled nodes, burn-in doubles as the leak-detection soak, so run it after the fill and pressure test. Watch coolant flow rates, supply and return temperatures, and delta-T alongside the electrical telemetry; a fitting that weeps only at temperature will not announce itself in a 20-minute test. Cooling behaviour at partial load deserves the same attention as full load, since AI halls spend their early lives lightly loaded — the effects are covered in the part-load problem in liquid cooling.

At the end of the soak, verify power draw per rack matches the expected load within 10%. Significant deviation indicates a misconfigured or failed component, not a rounding difference. Confirm cooling metrics — temperature, coolant flow rate — are stable under full load for at least four hours before moving on [14].

Step 4 — HBM integrity, ECC counters, and row remapping

HBM failures do not usually crash a job; they corrupt results or retire memory pages until the GPU is unusable. Two mechanisms matter, and both need a day-one baseline [2][5].

DRAM page retirement is the GPU's self-healing mechanism: memory pages that produce errors are retired from service and logged in InfoROM. NVIDIA's dynamic page retirement caps this at 64 retired pages — a card approaching that ceiling is functionally degraded regardless of how it looks on a bench, and the lifetime count is the odometer you should record on day one [2].

Row remapping replaces failing memory rows with spare rows. XID 63 means a remap was recorded successfully — managed degradation, with a reset window needed to activate it and recurrence to watch. XID 64 means the remap failed to record: the GPU cannot self-heal that address, and it is an RMA conversation [5].

ECC counters come in two flavours, and the distinction matters at acceptance: volatile counts reset when the GPU resets, while aggregate counts are lifetime cumulative. Baseline the aggregate counts on day one so later growth is unambiguous — a rising correctable-ECC count on one GPU while its peers stay flat is an early warning that beats waiting for a training run to crash [5].

The XID shortlist worth memorizing

Exit criteria for the memory sub-system, written before testing starts: zero uncorrectable errors and zero new row remaps in the final 48 hours of the campaign [6].

Step 5 — NVLink and the collective benchmarks that prove a fabric

Inside a node, GPUs talk over NVLink; across nodes they talk over InfiniBand or RoCE, usually with GPUDirect RDMA in the path. Both layers need proof, and the tool of record is NCCL — the collective communications library that NVIDIA's own stack uses for all-reduce, all-gather, and reduce-scatter [6][7][13].

Two local checks precede any multi-node test: confirm the on-board fabric works at full bandwidth between GPUs in the same chassis, verifying the device-to-device memory path and the NVLink topology as reported by the driver. Per-link state and per-link error counters should be read on SXM systems before any cluster test is attempted, because NCCL tests degrade or fail on top of a bad NVLink lane [7][13].

Then the cross-node sweep. The expected numbers, for large messages:

A practical acceptance target on a 400 GB/s fabric is roughly 92% of theoretical maximum — on the order of 370 GB/s on a large-message all-reduce [13]. Expect measured bus bandwidth near the platform's NVLink capability inside a node, and near the per-node NIC aggregate across nodes on a rail-optimized design [7][13].

Run it as a scaling ladder, not a single command

The most informative pattern is the same command repeated at increasing scope — one node, then two, four, eight, up to full cluster — sweeping message sizes from a few kilobytes to several gigabytes and recording large-message bus bandwidth at every rung. Three shapes tell you everything [7][15][16]:

• Flat curve — healthy fabric and correct rail wiring. Sign it.
• Cliff at a specific rung — the fault sits in whatever entered at that step: a leaf switch, a spine tier, or the inter-rack links. Bisect within that rung by testing node subsets [7].
• Uniformly low from two nodes — not a cable, a systemic setting: GPUDirect RDMA disabled, wrong topology detection, ACS enabled on the PCIe path, or congestion-control and PFC misconfiguration on an Ethernet fabric [7].

Small messages deserve one pass of their own. Sub-64 KB latency exposes switch and routing problems that large transfers hide, and it is exactly what gates small collectives during inference and mixture-of-experts routing [7].

Judge the fabric on bus bandwidth, not algorithm bandwidth. Algorithm bandwidth falls with node count by construction for all-reduce; bus bandwidth should stay roughly flat. If bus bandwidth itself sags with scale on a non-blocking fabric, that is a real finding [7].

Step 6 — Fabric soak, storage, and the power-cycle test

Once the ladder passes, the fabric gets 24 hours of continuous collective testing — looped all-reduce, all-gather, and reduce-scatter across the full cluster at both large and small message sizes. This heats the optics and stresses every link, which is the point: optics problems are thermal problems far more often than they are configuration problems [6].

Watch the port counters, not just throughput. Symbol errors, link-down events, and link recovery counts on any port should be effectively zero after the first day. A link that flaps under thermal load is a bad transceiver, a contaminated ferrule, or a damaged cable — and a link that trained below its rated speed or width is a defect even if it passes traffic [6][7].

Storage testing runs concurrently with GPU load, because that is the condition production will impose. Sequential and random I/O tests across every NVMe drive verify rated read and write throughput and catch drives with early-life failures — but a storage test that runs on an otherwise idle node is not evidence relevant to a training cluster. Add a checkpoint-shaped burst test: the write pattern a training job produces at a checkpoint boundary is nothing like a steady-state benchmark, and checkpoint stalls are a recurring cause of poor goodput [6][14]. Targets must be met under concurrent GPU load, which is one of the exit criteria [6].

Then pull the power. At least two deliberate cold shutdowns and restarts belong in the campaign, with a verification that every GPU, NIC, and drive returns: boot failures, missing devices, and devices that reappear with a changed PCIe width are all early-life findings that a steady-state soak will never produce [6][14].

Step 7 — A real workload, then a clean window

The final phase is 24 to 48 hours of an actual model run with checkpointing. This is the only step that combines compute, memory, fabric, storage, drivers, and the scheduler into one test, and it is where near-miss problems surface as step-time variance, NCCL timeouts, or storage stalls rather than clean errors. For a liquid-cooled hall, keep the coolant telemetry on the same timeline so a thermal event correlates with the workload evidence [6].

Then hold a mixed-load clean window of 48 hours that must be error-free to exit. A campaign that ends the moment the last script passes has no evidence that the fixes applied mid-campaign actually held [6].

Reading the exit criteria

The HPL spread criterion exists because of straggler research: on large clusters, a small number of slow nodes gate whole-job throughput. A node running 5–10% below the fleet median at identical settings usually has a throttling GPU, a link that trained below its rated width, or a cooling defect — and it is far cheaper to find now than in production [6].

Pass thresholds should come from the validated design and the exact installed bill of materials, not copied from another cluster. An arbitrary value imported from a different topology can hide real differences or manufacture false failures [16].

Step 8 — Snapshot the day-one baseline

Day one of a new cluster is the only time the hardware will be seen in a known-clean state, so record it. Without a baseline, day-two monitoring has nothing to measure drift against — and drift, not outright failure, is how most GPU problems announce themselves [5].

The per-GPU record worth keeping:

• Serial numbers, firmware and driver versions, vBIOS, and the full configuration inventory.
• DCGM diagnostic results as JSON, per level run, per GPU.
• ECC counters — aggregate and volatile — for both single-bit and double-bit.
• Row-remap status and lifetime page-retirement counts.
• Sustained clocks, power draw, and HBM temperatures under identical load.
• Per-GPU compute benchmark result, so performance outliers are visible from day one.
• Per-port fabric counters and the NVLink topology as built.
• A short benchmark run per GPU at identical settings, archived with the acceptance record [5][6].

That last item is the quiet one. Baselines that capture errors but not performance cannot expose a GPU that is 8% slower than its neighbours while technically healthy — and on a synchronized training job, one slow GPU drags the whole cluster. Record the performance baseline, not just the error history [5].

For liquid-cooled racks, the same baseline discipline applies to cooling: flow rates, supply and return temperatures, delta-T, and leak-detection sensor state recorded in the known-clean condition, so a later change is measurable rather than arguable. Our CDU maintenance guide covers what to watch once the loop is in service.

Putting the schedule on paper

Run as one campaign, the phases above add up to a campaign of roughly 124 to 176 hours — about 5.2 to 7.3 days per node or rack group [6]:

Two planning notes. First, the campaign is power-hungry by design — budget the electricity and the cooling capacity for it, and make sure the facility can supply and reject the load continuously for a week before scheduling it. Second, the schedule is per rack group, not per site: on a multi-rack AI factory, the gates layer outward from compute tray to rack to fabric to cluster, and each gate has its own exit criteria before the next begins [15].

The one thing not to do is to shorten the campaign because the hardware looks fine. A GPU that passes a four-hour test and fails in week three is not an unlucky draw — it is an untested component that behaved exactly as the bathtub curve predicts [8][10].

Questions to ask your integrator or deployment partner

1. What burn-in duration do you run, and what telemetry do you capture and hand over — per-GPU temperatures, power, ECC counters, fabric error rates? [9][10]
2. Which DCGM diagnostic levels run on every GPU, and is the JSON output archived per serial number? [1][5]
3. How is the multi-node NCCL validation structured — scaling ladder, message-size sweep, and what bus bandwidth do you commit to for our GPU count? [7][13]
4. How do you record and act on ECC and row-remap counters — what count triggers investigation, and what triggers replacement? [2][5]
5. What are the written exit criteria, and which of them apply to the final 48 hours specifically? [6]
6. Do you run at least two cold power cycles, and do you verify every device returns at its rated PCIe width afterward? [6][14]
7. How do you validate storage under concurrent GPU load, and how do you test the checkpoint write pattern? [6]
8. For liquid-cooled nodes, does burn-in double as the leak-detection soak, and what coolant flow, delta-T, and leak-sensor data is archived? [9]
9. What NVLink topology and firmware baseline do you confirm and freeze on the asset record? [15][16]
10. What is the quarantine and RMA path if a node trips a threshold mid-campaign — and does the clock restart for the replaced unit? [6][8]

Once the cluster is live, the same telemetry becomes operations infrastructure: GPU temperature, power, ECC, and fabric error trends are exactly what a monitoring platform has to carry to make drift visible. Our overview of what a compute center DCIM must cover maps those points onto the monitoring stack, and if you are sizing the electrical side to match, UPS battery backup sizing for AI data centers covers how the load's step behaviour changes the arithmetic.

Bottom line

Burn-in is not a formality before racking — it is the cheapest opportunity a GPU cluster owner ever gets to find a bad part. The sequence is well established: bring the rack up sequentially, freeze a firmware and configuration baseline, run DCGM diagnostics at level 3 on every GPU and level 4 at least once, soak thermally for 24–48 hours, prove NVLink and the InfiniBand fabric with a scaled NCCL ladder against a committed bus-bandwidth target, stress storage under concurrent GPU load, power-cycle twice, run a real workload, and hold a 48-hour clean window with zero uncorrectable errors, zero new row remaps, and zero link flaps. Record the day-one baseline — errors and performance — because every future diagnosis depends on it. The campaign costs about a week and, for an eight-rack AI pod, roughly 99,000 kWh of electricity. Discovering the weak GPU in week three of a production training run costs considerably more.

Planning a GPU deployment and need the infrastructure side validated too — liquid cooling loops, CDUs, power distribution, and white-space heat rejection sized for burn-in-class loads? Request a quote and we will scope the commissioning and cooling equipment your deployment needs.

References

1. dcgmi diag — DCGM active health diagnostics, run levels 1–4 and diagnostic test catalogue, NVIDIA DCGM documentation.
2. Dynamic Page Retirement — DRAM page retirement, the 64-page cap, and InfoROM logging, NVIDIA.
3. NVIDIA DCGM User Guide — health monitoring, telemetry fields, and diagnostics, NVIDIA.
4. AI Cluster Orchestration & Observability — XID error table (48, 63, 79, 92), DCGM diagnostic levels, and InfiniBand port counter reference, FlashGenius.
5. GPU Health on Day 1: DCGM Diagnostics, XID Errors and the Baseline to Record — the per-GPU baseline record fields, RDP GPU Mart.
6. Burn-In and Validation for GPU Clusters: Catching Failures Before Production — phase schedule, 72 h minimum, exit criteria, straggler effects, RDP GPU Mart.
7. NCCL-Tests Validation: Bus Bandwidth Targets for AI Fabrics — scaling ladder, curve shapes, small-message latency, tolerance, RDP GPU Mart.
8. GPU Burn-in Testing: Definition & Process — bathtub curve, 48–72 h detection rate, thermal saturation, power delivery validation, Inference Systems.
9. Pre-Ship Testing for AI Server Racks: Burn-In, Inspection, and Transit Surveillance — four complementary test stages, burn-in cell heat rejection, SpecForge.
10. What Is Burn-In Testing? — infant mortality, bathtub curve, and simultaneous stress of power, cooling, memory, and interconnects, Leviathan Systems.
11. NCCL Bandwidth Validation: Proving a GPU Fabric Before Production — NVL72 NVLink rates, NDR400 ports, marginal MPO connector failure mode, Leviathan Systems.
12. Validating Multi-Node GPU Clusters with NCCL Tests — expected bus bandwidth by GPU and interconnect, link-state prerequisites, Saturn Cloud.
13. A practitioner's guide to testing and running large GPU clusters for training generative AI models — NVLink, RDMA, NCCL at 92% of theoretical, storage with fio, Together AI.
14. GPU Server Rack and Stack: Hardware Deployment in Colocation Facilities — staggered power-on, BIOS and firmware baseline, 24–72 h burn-in, power and cooling verification, RAX.
15. How to Design and Validate an NVIDIA GB200 or GB300 NVL72 AI Factory — validation gates from rack bring-up to cluster acceptance, with exit criteria, s0x.org.
16. How to Design and Validate an NVIDIA GB200 or GB300 NVL72 AI Factory — layered NCCL test matrix, inference acceptance metrics, evidence chain, Digital Thought Disruption.
17. NVIDIA GB200 NVL72 — 72-GPU NVLink domain, 130 TB/s NVLink switch bandwidth, per-GPU interconnect rates, NVIDIA.