Lifetime ECC Double-Bit Errors: What nvidia-smi Can Tell You About a GPU's Memory

November 1, 2026|By Assurd Engineering Lab|The Playbook

Graphics memory holds billions of bits, and once in a while one of them flips. On most graphics cards nobody ever knows. On cards with error-correcting (ECC) memory, the memory controller notices, repairs what it can and counts what it finds. Those counts are some of the few records a GPU keeps of its own history.

This post explains what the counts mean and how an Assurd certification uses them (SOP-GPU-001 v1.1, Step 4.4, as amended on 2026-11-01). The short version: when the card reports the counters, the lab records them, and too many uncorrectable errors stop the certification. When the card does not report them, the certificate says so.

How error correction works

ECC stores a few extra check bits with every block of data. When the block is read back, the controller recomputes the check bits and compares. A mismatch shows that something changed, and its pattern can show which bit changed.

The textbook scheme is called SEC-DED: single error correction, double error detection. To pinpoint any one flipped bit among $m$ data bits and $r$ check bits, the $r$ check bits need enough patterns to name every bit position, plus "no error":

2^{r} \ge m + r + 1 .

For a 64-bit word, $r = 6$ is not enough ( $2^6 = 64 < 71$ ) and $r = 7$ is ( $2^7 = 128 \ge 72$ ). One extra parity bit across the whole word then tells a single flip, which can be corrected, from a double flip, which cannot. That gives the classic 72-bit word: 64 data bits plus 8 check bits, a 12.5 % overhead.

GPU memory controllers use their own codes, which differ by product and memory type. The counters the driver reports do not depend on those details. They count two outcomes:

Correctable (single-bit) errors. The controller fixed the data. The program never saw the error.
Uncorrectable (double-bit) errors. The controller detected the error but could not repair it. The data could not be recovered.

A steady trickle of correctable errors is what ECC is for. Uncorrectable errors are a different matter: each one is a read that returned data the card knew to be wrong.

Volatile and lifetime counters

The driver keeps two sets of counters:

Volatile counters cover the time since the driver last loaded. They reset at reboot.
Aggregate counters are the card's lifetime totals.

A volatile count of 0 from a freshly booted bench says nothing about the card's past. Only the aggregate (lifetime) counters are used.

Which cards report ECC

ECC counters are reported by cards with ECC memory: typically workstation and data-centre cards. Most consumer cards do not report them, and their counters read N/A. A few high-end GeForce cards (the RTX 3090 Ti and RTX 4090, for example) offer an optional ECC mode in the driver; it is off by default, costs some usable VRAM, and the card reports counters only while it is on. N/A is not a count of zero; it means the card has no counters to report.

The check therefore applies when the card reports ECC counters, on every tier. No tier requires a card to report them, and a card without them is not penalised. The report comes from NVIDIA's nvidia-smi tool, so this check covers NVIDIA cards only.

Reading an nvidia-smi report

At the end of Phase 4 the technician saves the driver's ECC report to a text file and uploads it (optional):

nvidia-smi -q -d ECC > ecc.txt

The server archives the report privately and reads the lifetime counters from its Aggregate section. Current drivers split them by memory type. This excerpt is from the layout the parser is tested against; the counts are made up, and the spacing is shortened:

Aggregate
  SRAM Correctable           : 3
  SRAM Uncorrectable Parity  : 1
  SRAM Uncorrectable SEC-DED : 2
  DRAM Correctable           : 412
  DRAM Uncorrectable         : 9

The lifetime uncorrectable count is the sum of the "Uncorrectable" lines, and the correctable count the sum of the "Correctable" lines:

\begin{gathered} \mathrm{DBE} = 1 + 2 + 9 = 12 \\ \mathrm{SBE} = 3 + 412 = 415 \end{gathered}

The sum includes errors in the GPU's on-chip SRAM as well as in the graphics memory (DRAM). So the certificate's "double-bit" figure is the driver's total of uncorrectable errors, a little broader than the name suggests.

Older drivers print the same information as "Single Bit" and "Double Bit" blocks, each with a "Total" line; the parser reads those totals.

Four rules keep the reading honest:

A line reading N/A is left out. If every line is N/A, the count is null, never 0, and the certificate reads "ECC counters: Not reported by this card".
Volatile counters are ignored.
A report listing more than one GPU is refused: the bench holds only the card under test.
A file that is not an ECC report is refused rather than read as zero.

The Hard Limit

\mathrm{DBE}_{\text{lifetime}} > 100 \ \Rightarrow\ \text{Hard Limit}

More than 100 lifetime uncorrectable errors is a Hard Limit: a VRAM integrity failure. The card is quarantined, and the flag cannot be overridden. Any count from 0 to 100 is recorded and shown on the certificate, with no limit.

The comparison is strict. 100 errors pass; 101 do not.

Report	DBE	SBE	Result
Consumer card, all `N/A`	null	null	Not reported
Current layout (above)	12	415	Shown
Older layout	101	58	Hard Limit

All three are test reports in the drivers' layouts, with made-up counts.

What the counters cannot tell you

The counters are useful precisely because they are simple. They also have limits:

No count is not a clean bill of health. A card that reports N/A tells you nothing about its memory, in either direction.
They have no timestamps. A lifetime total does not say whether the errors came last week or years ago, or in a burst or a trickle.
They are counts kept by the card and its driver. The lab reads them; it does not measure the memory independently. The certificate shows them as the driver reported them.

The memory is also exercised in other parts of the certification. On Premium certifications, Stable Diffusion XL throughput well below the SKU reference with an unsteady iteration speed is a Soft Limit, and the SOP asks the technician to cross-reference the ECC counters when the card reports them (Step 4.3).

Calibration status

The 100-error limit is an engineering prior pending fleet calibration (n ≈ 300). It is not fitted to failure data, and we do not describe it as empirical. Once enough cards with ECC counters have been certified, the limit will be reviewed against their counts, and any change will be versioned in the SOP.

See it on a certificate

On a certificate, the Silicon DNA tile shows the lifetime ECC counters, or "ECC counters: Not reported by this card". The sample certificate shows the second: its ECC field is empty, as on a consumer card. It is illustrative: no card was measured.

The other Phase 2 identity check, the firmware lookup, is explained in VBIOS SPI Dumps, Hashing and Silicon Identity.

Notation

Symbol	Meaning	Unit
$m$ , $r$	Data bits and check bits in a code word	bit
DBE	Lifetime uncorrectable (double-bit) errors	count
SBE	Lifetime correctable (single-bit) errors	count