Silent Data Corruption: The Failure That Passes Every Test

A processor can pass wafer sort, final test, system-level test and burn-in, go into a server, and then return a wrong answer from an ordinary instruction. No machine check fires and no log line appears. The program carries on with a bad number. Meta described one such case in which a CPU core calculated a file size as zero, and files went missing from a database as a result.

Google, Meta and Alibaba have now published enough data to show that these cases are a measurable population of chips and not curiosities. This post sets out what they have reported, why the defects get through, and what chip designers and test engineers can do about it.

Key takeaways

• Published rates cluster around a few hundred to about a thousand affected chips per million. Alibaba measured 3.61 per 10,000 CPUs. Google estimates about 1,000 defective parts per million (DPM) for chips that cause silent corruption, and calls that a lower bound.
• Newer chips have not made this go away. Alibaba found no drop in failure rate with newer microarchitectures, and Google reports no material improvement across four platform generations from 22nm to 5nm.
• Part of the population cannot be caught at the factory, because the chip passed when it was tested. Google attributes 29% of the chips it sent back for analysis to early-life failure.
• Detection delay is the strongest lever in our model. Google sees in-field scan as the most principled hardware route, though the right patterns are still an open question.
• Google argues that today's fault metrics fit these defects poorly, so a high coverage figure offers little comfort.

What the hyperscalers have reported

These figures come from different fleets, test content and definitions, so read them as a range.

Source	Population	Reported figure	Read it with care because
Meta, February 2021	Hundreds of thousands of machines	Hundreds of CPUs found with silent errors	An order of magnitude, with no exact rate
Google, May 2021	Google's server fleet	A few "mercurial" cores per several thousand machines	Google states that it cannot disclose exact rates
Alibaba and Tsinghua, October 2023	Over one million CPUs from one manufacturer, tested over 32 months	3.61 per 10,000 CPUs, ranging from 0.082 to 9.29 per 10,000 by microarchitecture	Limited by the test suite, which the authors say missed some known cases
Google, August 2025	11 platforms over 4 generations, 22nm to 5nm	About 1,000 DPM for chips that cause silent corruption, ranging from 318 to 1,857 by platform	A stated lower bound, and older platforms have had longer in service
Meta, July 2025	Meta's fleet, discussing AI hardware	About one fault per thousand devices	A summary figure from a blog post

Google's 2025 paper adds a wider figure: it estimates that test escapes of all kinds cause about 5,000 DPM of chips to be swapped over their life, against an industry expectation of 100 to 500 DPM. That gap is the "10×" in its title. It is one large customer's estimate.

Three reported examples show the range of symptoms:

• Meta. Scala's math.pow, computing Int(1.1^53), always gave 0 on Core 59 of one CPU. The correct integer part is 156. Int(1.1^52) gave the right answer, 142. The zero became a file size, so the decompression step skipped the file.
• Google. One AES fault was "self-inverting". Encrypting and decrypting on the same core worked perfectly, while decrypting anywhere else produced gibberish.
• Alibaba. A storage application kept reporting checksum mismatches. Weeks of debugging traced them to a checksum-related instruction on one processor.

Why these chips pass

Time on the test floor. Google notes that more than several minutes of test time per chip is usually seen as impractical, while the system-level tests that find these defects can run for hours.

Metrics that do not match the physics. Scan patterns are generated against stuck-at, transition, small delay and cell-aware models. Google's paper argues that these models fit real defects poorly, and that much of what scan catches today is caught by luck. It cites an estimate that for timing-independent defects, over 90% of detections cannot be explained by the values the fault metric asked for.

Marginal and context-dependent behaviour. Many of these defects misbehave only at certain operating points or in certain execution contexts. Google has seen lower frequency sometimes increase the failure rate. Alibaba found that in some settings SDC frequency grows exponentially with temperature, even inside the allowed range. The Stanford and Google ITHICA paper reports that the same instruction with the same inputs in the same thread can give different outputs depending on execution context. Of the defective servers it analysed, only one reproduced with single-instruction tests.

Time. Google reports that a significant share of escapes are early-life failures, despite hardware burn-in, and that they appear well before the chip's expected end of life.

Diagnosis is the weak link. Google sends fewer than 10% of detected parts back to vendors, and it picks the interesting ones, so this is not a random sample. The outcomes it reports were:

Outcome at the vendor	Share of returned chips
No trouble found	36%
Early-life failure (now fails tests it once passed)	29%
Test gap, new test developed	18%
Test gap, no successful test yet	10%
Damaged on the way to the vendor	7%

More than a third of the returned chips taught the vendor nothing about its test programme.

A worked example: what detection delay is worth

Every input is an illustrative assumption for a hypothetical fleet, and the results are not a prediction for any real one. The defect rate and error rate are chosen to be of the same order as Google's figures.

Assumption	Value
Fleet	100,000 CPUs, 64 cores each, so 6,400,000 cores
Chips that become SDC-causing over a 5-year life	1,000 DPM, so 100 chips, or 20 new ones per year
Bad cores per bad chip	1, a simplification, since Alibaba found about half of its faulty CPUs had all cores affected
Wrong results per bad chip per day	0.02
Probability that one test pass catches a bad chip	50% unless stated

On those inputs, lifetime incidence is 100 in 6,400,000 cores, or about 15.6 per million cores. The 0.02 rate is rounded from Google's median of 820,000 incorrect outputs per billion chip hours, which is 0.00082 per hour, or 0.0197 per day. That median was measured while checking known-bad CPUs under system-level tests, so a production workload could be well above or below it.

If a chip turns bad at a random time and tests run every T days, each with catch probability p, the average time it stays in service is T × (1/p − 0.5). Multiply that by the arrival rate and the error rate to get wrong results per year.

Scenario	New bad chips per year	Average days in service	Bad chips live at any moment	Wrong results per year
A. Test every 180 days	20	270	14.8	108
B. Test every 60 days	20	90	4.9	36
C. Test every 15 days	20	22.5	1.2	9
D. Scenario C, with p raised to 90%	20	9.2	0.5	3.7
E. Scenario C, with half caught before deployment	10	22.5	0.6	4.5

To check scenario A: 180 × (2 − 0.5) = 270 days, 20 × 270 ÷ 365 = 14.8 live chips, and 20 × 270 × 0.02 = 108 wrong results. That is about 2.3 bad cores per million at any moment, which is why the problem looks so rare from inside any one job.

Going from a 180-day to a 15-day interval cuts the exposure twelvefold. Raising the catch rate from 50% to 90% cuts it about 2.5 times. Stopping half the bad chips before deployment cuts it by half. The model counts every wrong result as equal, though one bad metadata write can cost far more than many wrong pixels.

What chip designers and test engineers can do

This is our reading of the published work. Overheads and gains are the authors' claims.

Stage	Action	Evidence and limits
Architecture	Plan for in-field scan from the start: patterns held in off-chip flash, delivered over high-speed I/O, and updatable after shipping	Google names Amazon's Deterministic In-Fleet Scan Test, Intel's In-Field Scan and NVIDIA's In-System Test as products of this kind. It quotes earlier research at about 1% area, 1% power and 3% performance in online mode
Power and clock design	Provide on-chip voltage and clock control usable during field test	No external tester is available in the field, and temperature control there remains hard
Microarchitecture	Protect selected logic with residue or parity checks where the critical blocks can be justified	Google puts duplication at over 100% area and energy overhead for the blocks covered, and parity often above 20%. No validated fault model yet ranks blocks by defect risk
Pattern generation	Trial newer metrics such as PEPR, and spend the longer field test budget on them	PEPR may need 10 to 100 times more patterns, and so far addresses only timing-independent defects
Test conditions	Generate delay patterns per operating voltage, since the longest path through a gate can change with voltage	Google notes that ATPG tools rarely analyse test conditions fully
System-level test	Run hot, run all cores together, and include multi-threaded content	Alibaba's consistency defects could only be found with multi-threaded tests, and they accounted for 8 of the 27 CPUs it studied closely
Test content	Use field history to prioritise, and use long, varied programs with instruction-level checks	In one Alibaba environment, 560 of 633 test cases found nothing. Reordering cut one regular round from 10.55 hours to 1.02 hours with better coverage on its faulty set. ITHICA reports 39% more defective servers than the programs' native output checks and 69% more than SiliFuzz, in a pool of suspect servers
Diagnosis	Add hooks that shorten error detection latency, and agree a return flow with your largest customers	Latency can reach billions of cycles, and vendors often cannot run a customer's workload
Data exchange	Use the OCP formats for test inputs, outputs and part history, and rate tests by efficiency	OCP defines test efficiency as the inverse of time to failure. Its listed tools include OpenDCDiag (Intel), Open Field Health Check (AMD) and DCGM (NVIDIA)

One warning: Google says system-level suites have looked complete, with detections falling away, only for further testing to find many more escapes. A falling detection rate does not prove a clean fleet.

What the fleet side shows you

On one large-volume platform, Google found 12% of defective machines in pre-deployment testing, 29% in online and offline testing in service, 49% through system health and forensics, and 10% through checks inside user software.

Alibaba found 90.36% of its faulty CPUs before production, but counted only what its test suite caught, so the two splits are not comparable.

Forensics is cheap. Google's rule flags a core if at least 80% of a machine's kernel crashes land on it, with at least 5 crashes and 3 or more distinct top-of-stack symbols in 30 days. Google says over 70% of flagged cores proved SDC-causing and under 10% were false alarms. Meta claims its Hardware Sentinel, which analyses kernel-space exceptions without test time, outperforms testing-based methods by 41%.

Meta's 2022 data shows why test modes get combined. Maintenance-window testing reached a machine about every 180 days, while short tests beside live workloads reached 70% of the common coverage in about 15 days. Each found defects the other missed: 23% and 7% respectively. By 2025 Meta said maintenance-window testing covered the fleet every 45 to 60 days.

Standard data protection helps less than assumed. Alibaba points out that if the CPU computes a wrong value before a checksum or ECC is generated, the check simply protects the wrong value. Checksum code also leans on vector units, one of the features it found most vulnerable. And 99.9% of the double-precision errors it measured were below 0.02%, often too small for a range check to notice.

Where should your effort go?

Answer these five questions before the next test plan or architecture review:

1. Will your chip run where a wrong answer can spread? Data centre compute, storage controllers and AI training all qualify. If not, the cost case for field test is weaker.
2. Can you still change the DFT architecture? If yes, in-field scan with updatable patterns is the first thing to price, since it cannot be added later.
3. Do you get field returns with real diagnosis? If not, fix that before buying more test time. Without feedback you cannot tell which new test is working.
4. Do you test across the voltage, frequency and temperature range that customers use? If your screens run at a few fixed points, adding conditions may be the cheapest step.
5. Is your test content distinct from your customers'? Google warns that similar content on both sides makes escapes harder to estimate. Ask them what they find that you do not.

Our SoC lifecycle on The Semiconductor Hub shows where these DFT and test decisions fall in a programme.

FAQ

Are these errors caused by cosmic rays?

Mostly not, on the published evidence. Soft errors are transient, while these defects are tied to specific chips and keep recurring there, at rates Google describes as much higher than radiation-induced ones. The OCP AI white paper does treat transient faults as one source of SDC.

Is this only a CPU problem?

No. Most fleet data covers CPUs, but Google reports a similar picture across compute chip types. Its median for known-defective ML accelerators on training workloads was 916,000 incorrect outputs per billion chip hours of checking. OCP's workstream covers CPUs, GPUs and other accelerators.

Can you just run everything twice?

For the most critical code, yes, and Google does double or triple some computations. As a general answer it costs too much, because detection roughly doubles the work and correction may triple it.

Sources

• Hochschild et al. (Google), "Cores that don't count", HotOS '21, May 2021.
• Dixit et al. (Facebook), "Silent Data Corruptions at Scale", arXiv:2102.11245, 22 February 2021.
• Dixit et al. (Meta), "Detecting silent data corruptions in the wild", arXiv:2203.08989, March 2022, and the Meta Engineering post on silent errors, 17 March 2022.
• Wang et al. (Tsinghua, Ohio State, Alibaba Cloud), "Understanding Silent Data Corruptions in a Large Production CPU Population", SOSP '23, October 2023.
• Open Compute Project, "Server Component Resilience", Revision 1.0, 27 September 2024.
• Meta Engineering, "How Meta keeps its AI hardware reliable", 22 July 2025.
• Mitra et al. (Google), "Silent Data Corruption by 10× Test Escapes Threatens Reliable Computing", arXiv:2508.01786, August 2025.
• George et al. (Open Compute Project), "Silent Data Corruption in AI", white paper, 27 August 2025.
• Vavelidou et al. (Stanford, Google), "ITHICA: Intra-Thread Instruction Checking Approach for Defect-Induced Silent Data Corruptions", arXiv:2605.15638v2, 29 September 2026, to appear at MICRO 2026.

The worked example uses illustrative numbers. Rates, overheads and detection gains are the companies' and authors' own claims, as published on the dates shown.