Your first hardware design will probably fail in the field because it was proven under lab conditions: stable supply, room temperature, short test runs and well-behaved neighbours. The field adds supply transients, heat, vibration, ageing, noisy partner devices and years of run time, and it exposes every state, register and corner case the design never planned for. You prevent most of these failures by designing for recovery, for observability and for the worst case, and then by testing long enough and harshly enough to prove it.
"Everyone should get a lecture on why their first design will not work in the field."
The journey from lab prototype to field deployment is full of invisible pitfalls. What works beautifully under controlled conditions for a few minutes can fail badly when it meets real-world stress. Whether the cause is an environmental stressor, an overlooked design flaw or incomplete testing, first-generation designs rarely survive long-term deployment without significant iterations. This post, part of a multipart series, looks at why your first design will fail in the field and, more importantly, how to design for reliability from the start.
Key takeaways
- Every state machine needs a defined path out of every possible state code, including the ones you believe are unreachable, and synthesis must be told to keep that logic.
- A watchdog is the last line of defence, not the design for recovery. Design the recovery first, then add the watchdog.
- If you cannot read a register in a deployed unit, you cannot debug it. Plan debug access and field telemetry at architecture time.
- Treat every input from another device as untrusted: check range, timing and sequence before acting on it.
- Decoupling is sized from a target impedance and checked with power integrity analysis, not sprinkled by habit. Ceramic capacitors lose real capacitance with DC bias, temperature and age.
- A one-in-a-million event happens every day across a large installed base. Rare corner cases deserve the same priority as common ones.
1. The forgotten "unreachable" state: ghost states that lock up systems
State machines govern most digital systems and keep the flow between conditions predictable. In the real world, though, disruptions such as clock glitches, supply dips or radiation-induced bit flips (single event upsets) can push the state register into a code the designer never assigned. A 3-bit one-hot machine with three legal states has five illegal codes. Without a defined recovery path the system can freeze, drive wrong outputs or crash entirely.
Example
In automotive microcontrollers, supply transients during engine cranking can push state machines into undefined states. If an ECU (electronic control unit) locks up during start-up, critical functions such as ABS or traction control may be unavailable. This is why automotive supply transients are specified and tested in standards such as ISO 7637-2 and ISO 16750-2, which define cranking dips and load-dump pulses that the electronics must survive.
Fix
- Ensure every state code has a defined transition, including the unreachable ones. A
defaultbranch that returns to a known safe state is the minimum. - Implement hardware watchdog timers to force a reset when behaviour is undefined, and make sure the watchdog cannot be serviced from inside the stuck loop.
- Use fault-tolerant FSM design, where illegal states are redirected to a known safe state and the event is flagged, so firmware or a log can see that it happened.
- Check what synthesis did. Many tools treat the default branch of a fully decoded case statement as unreachable and remove it. Use your tool's safe-FSM option or attribute (the exact name differs between vendors) and confirm in the synthesis report that recovery logic survived.
Worked example: a one-hot FSM that recovers from an upset
The module below has three legal one-hot states. The default branch catches any other code, returns to IDLE on the next clock and pulses fault. The testbench forces a two-hot code to model an upset and checks that the machine recovers in one clock.
// Three-state controller with one-hot encoding and explicit illegal-state recovery
module safe_fsm (
input wire clk,
input wire rst_n,
input wire start,
input wire done,
output reg busy,
output reg fault // high while an illegal state is present
);
localparam [2:0] IDLE = 3'b001,
RUN = 3'b010,
STOP = 3'b100;
reg [2:0] state, next;
always @(posedge clk or negedge rst_n)
if (!rst_n) state <= IDLE;
else state <= next;
always @(*) begin
next = IDLE; // default: any unlisted code returns to IDLE
fault = 1'b0;
case (state)
IDLE: next = start ? RUN : IDLE;
RUN: next = done ? STOP : RUN;
STOP: next = IDLE;
default: begin next = IDLE; fault = 1'b1; end
endcase
end
always @(*) busy = (state == RUN);
endmodule
module tb;
reg clk = 0, rst_n = 0, start = 0, done = 0;
wire busy, fault;
safe_fsm dut (.clk(clk), .rst_n(rst_n), .start(start), .done(done), .busy(busy), .fault(fault));
always #5 clk = ~clk;
initial begin
#12 rst_n = 1;
#10 start = 1; #10 start = 0;
#20 done = 1; #10 done = 0;
#10 force dut.state = 3'b110; // model an upset: two hot bits
#1 $display("t=%0t upset state=%b fault=%b", $time, dut.state, fault);
#4 release dut.state;
@(posedge clk); #1 $display("t=%0t after one clock state=%b fault=%b", $time, dut.state, fault);
if (dut.state !== 3'b001) $display("FAIL"); else $display("PASS: recovered to IDLE");
$finish;
end
endmodule
Simulation output: the upset is seen at t=73 with fault=1, and after the next clock edge the state is 001 (IDLE) with fault=0. In a real design you would latch fault into a sticky status register so the event is visible later, which leads to the next failure mode.
2. Inaccessible internal registers: debugging nightmares
Some internal registers in a chip may become corrupted, stale or misconfigured. If the design gives no way to read or modify them from outside, debugging is close to impossible, and engineers are left guessing at the root cause or resetting the whole system and hoping.
Example
A consumer-grade camera sensor intermittently stops working after several months. Engineers later discover that an internal temperature compensation register is stuck because of an undefined register refresh cycle. Because the register was not externally accessible, debugging required microscopic probing of the silicon, which takes weeks and destroys the sample. (This is an illustrative scenario, not a specific product.)
Fix
- Always include a debug interface, such as JTAG (IEEE 1149.1), a vendor debug port or a UART diagnostic mode, and decide how it is secured in production rather than removing it.
- Provide a way to read and write critical internal registers externally, even after deployment.
- Implement periodic self-checks of register integrity: read back configuration, compare against a stored copy or CRC, and rewrite or flag on mismatch.
- Add sticky status bits and event counters (illegal FSM state seen, watchdog resets, ECC corrections, brown-out events) so that a returned unit can tell you what happened before it failed.
3. Hidden dependencies: what if another chip controls yours?
Your chip may be perfectly designed, but what if it relies on signals from an external device that has undocumented quirks or firmware bugs? When that device glitches, your chip will behave unpredictably, and the field report will name your part, not theirs.
Example
In industrial automation, a communication controller relies on an external FPGA for signal processing. The FPGA intermittently fails because of overheating and sends invalid control signals to the controller, which then misinterprets sensor data.
Fix
- Implement signal integrity and plausibility checks: if incoming data is out of range, out of sequence or late, ignore it or flag it.
- Protect messages between devices with a CRC, a sequence counter and a timeout, the same pattern used for safety-relevant communication in automotive systems.
- Allow emergency manual overrides for external control signals, and define what your device does when its partner goes silent.
- Always validate dependencies, including power-up order and reset timing: your design does not exist in isolation.
4. The cruel analog world: not enough decoupling capacitors
In the digital domain, 1s and 0s look reliable, but the analog reality underneath is not. Insufficient or badly placed decoupling leads to supply droop, ground bounce and noise that turns into random logic failures over time.
Example
A high-speed microcontroller fails sporadically in the field, despite working flawlessly in the lab. Engineers later find that supply noise was corrupting signal integrity because of inadequate decoupling capacitors, causing random logic failures.
Worked example: target impedance
A common starting point for power distribution design is a target impedance: the allowed ripple divided by the transient current. The numbers below are illustrative.
| Quantity | Value | Note |
|---|---|---|
| Core supply | 1.0 V | Illustrative |
| Allowed ripple | 3% = 30 mV | From the device data sheet |
| Transient current step | 2 A | Worst-case load step |
| Target impedance | 30 mV / 2 A = 15 milliohm | Must hold across the frequency band of the load step |
The bulk capacitors, the ceramic capacitors and the package and die capacitance each cover part of the frequency range. A ceramic capacitor is only useful up to its self-resonant frequency, which depends heavily on mounting inductance, so a capacitor far from the pin, on long traces or a single via, may do very little. That is why placement matters as much as value.
Fix
- Use proper capacitance values in the right locations, close to the power pins with short, low-inductance connections. Do not just sprinkle capacitors randomly.
- Perform power integrity analysis (impedance versus frequency, and transient droop) to find noise risks before layout is frozen.
- Account for capacitor derating and ageing. Class 2 ceramics such as X7R and X5R lose a large share of their nominal capacitance under DC bias and lose more slowly with age; electrolytic capacitors dry out faster at high temperature. Check the manufacturer's DC-bias and ageing curves, not just the nominal value.
5. The lab is a lie: did you test in real-world conditions?
Your prototype may run perfectly in a temperature-controlled lab, but real conditions add thermal drift, humidity, vibration, condensation and electromagnetic interference.
Example
A consumer drone has flawless flight stability in indoor lab tests. Outdoors, however, temperature changes in key sensor circuits cause small voltage shifts, leading to altitude miscalculations.
Worked example: how long is "long enough"?
Accelerated life testing raises temperature so that weeks in a chamber represent years in use. The Arrhenius model gives the acceleration factor AF = exp[(Ea/k)(1/T_use - 1/T_stress)], with temperatures in kelvin and k = 8.617e-5 eV/K. With an illustrative activation energy of 0.7 eV, a use temperature of 55 C (328 K) and a stress temperature of 125 C (398 K), AF is about exp(8124 x 0.000536), roughly 78. A 1,000-hour test then represents about 78,000 hours, close to nine years, for failure mechanisms that follow that activation energy. Mechanisms with a different activation energy, or ones driven by humidity, vibration or thermal cycling, need their own tests.
Fix
- Conduct long-term stress tests: run prototypes for weeks or months to uncover hidden failures, and use accelerated tests to reach field-equivalent life.
- Perform multi-environment testing, including extreme temperatures, temperature cycling, humidity and vibration. For automotive ICs, AEC-Q100 defines the qualification stress tests suppliers are expected to pass.
- Simulate real-world interference, such as electromagnetic noise from industrial machinery, and run EMC immunity tests early rather than only at certification.
6. Ignoring boundary cases: rare bugs are still catastrophic
A bug that occurs only once every six months might seem trivial, until it happens to thousands of units in the field. With 100,000 units deployed, a once-every-six-months event per unit means more than 500 occurrences a day across the fleet. Neglecting corner cases leads to silent failures that make products unreliable.
Example
A networking device experiences packet loss only during a leap year because of a faulty date-checking algorithm. Engineers dismissed it as "low priority" during development, but when thousands of units fail on the same day, the manufacturer faces serious reputational damage. Date bugs of this kind are well documented in practice: the first-generation Microsoft Zune 30 GB players froze on 31 December 2008, the last day of a leap year, because of a day-counting loop in a clock driver.
Fix
- Treat rare scenarios with the same priority as common ones, and estimate their fleet-wide frequency before ranking them.
- Perform exhaustive validation of timing edge cases: counter roll-over, leap years, time zone changes, simultaneous events and back-to-back requests.
- Run simulations and tests at extreme operating conditions to uncover latent bugs, and use formal property checking for control logic where simulation cannot reach every case.
Common mistakes reviewers look for
| Mistake | Why it fails in the field | What a reviewer asks |
|---|---|---|
| Default branch removed by synthesis | Upset leaves the FSM stuck in an illegal code | Show me the safe-FSM setting and the netlist check |
| Watchdog kicked from a timer interrupt | Main loop hangs but the watchdog is still serviced | Where is the kick, and what proves the task is alive? |
| No sticky fault status | Returned units carry no evidence | What does a returned unit tell us about its last fault? |
| Partner device trusted blindly | Its bug becomes your field failure | What happens if this input is stuck, late or out of range? |
| Decoupling chosen by nominal value | Real capacitance far lower under bias and temperature | What is the effective capacitance at the operating voltage? |
| Tests only at room temperature | Drift and timing margins collapse at extremes | Which tests ran at minimum and maximum temperature and supply? |
Designing for reality, not just the lab
A flawless lab prototype does not mean a flawless product. The real world is messy, unpredictable and sometimes plainly cruel to hardware engineers. First designs often fail because they assume ideal conditions instead of designing for the worst case.
Best practices for field-proof design:
- Build a robust failure recovery mechanism: watchdogs, redundancy and fail-safe logic.
- Test beyond normal conditions: stress the system the way real users and real environments will.
- Prepare for environmental unpredictability: power fluctuations, EMI and thermal expansion.
- Ensure debugging access: if you cannot inspect faults, you cannot fix them.
Your lab might love your chip, but the field will push it to its limits. If you want products that endure, design with reality in mind, not just theory.
FAQ
Is a watchdog timer enough to make a design field-safe?
No. A watchdog recovers from a hang by resetting, which may itself be unsafe or lose data. It is a backstop. The design should first detect and recover from faults locally, such as illegal states, bad inputs and corrupted registers, and only fall back to a reset when that fails.
Why would synthesis remove my default case?
If every reachable state is decoded, the tool can prove that the default branch is never taken in normal operation and optimise it away. Upsets are not part of that analysis. Safe-FSM options tell the tool to keep recovery logic for unused codes.
How long should field-representative testing run?
Long enough to cover the target life for each failure mechanism, usually through accelerated tests (temperature, humidity, cycling) combined with weeks of real-condition running for system-level issues. The Arrhenius example above shows how test hours map to field years for temperature-driven mechanisms.
What should debug access look like in a production part?
Keep it, but control it: lock it behind authentication or fuses, and provide a read-only diagnostic path for status and counters so field engineers can collect evidence without opening security holes.



Your comments will be moderated before it appears here.