TDEB Top Bar

Trending
Semiconductor Industry News — The Digital Electronics Blog
Follow Us

TDEB Masthead

The Semiconductor Hub

Semiconductors.  Chips and Systems.  Innovation.

Indexed by leading Search Engines, AI and referenced by Engineers worldwide 500+ Articles 10M+ Pageviews 30+ Reports 50+ Citations

The Art of Debugging: Make It Fail

mg
by
0
Share Facebook X LinkedIn WhatsApp Email

Debugging is one of the most valuable engineering skills, yet it is rarely taught in any formal setting. Most engineers learn it the hard way, by trial, error, experience, and long days and nights. TechOnline published three excerpts, titled "Make It Fail", from David J. Agans's book Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems (AMACOM Books, 2002). The book is an informative and logical introduction to the skill and art of debugging electronic circuits and systems.

Where to read the excerpts and the rules

TechOnline has since closed, and only the third excerpt survives, as an Internet Archive copy of "Make It Fail: Part 3". The first two parts were not archived. The author's site still offers Chapter 2, "The Rules, Suitable for Framing" (PDF), which lists all nine rules, and an about-the-book page.

The nine rules

As listed in the Chapter 2 excerpt, the rules are:

  1. Understand the system
  2. Make it fail
  3. Quit thinking and look
  4. Divide and conquer
  5. Change one thing at a time
  6. Keep an audit trail
  7. Check the plug
  8. Get a fresh view
  9. If you didn't fix it, it ain't fixed

The list is short enough to tape to a lab bench, which is exactly what the book suggests.

Why "make it fail" comes so early

You cannot fix what you cannot see. A bug that shows up once a week on a customer's board is almost impossible to study. Once you can make it appear on demand, you can look at it, measure it, change one thing and see whether it goes away. Being able to reproduce a failure also gives you the test you need at the end: if the failure no longer appears under the same conditions, you have real evidence that the fix worked.

In practice, making something fail means writing down exactly what was happening when it failed (inputs, temperature, supply voltage, firmware version, traffic pattern), then recreating those conditions in a controlled setup. For intermittent problems, it often means making the conditions harsher so the failure happens more often, for example raising the clock rate, lowering the supply voltage, or running the test in a loop overnight.

How the rules fit together

  • Understand the system and make it fail set up the problem.
  • Quit thinking and look and divide and conquer find where the fault is, by observing the real behavior and narrowing the search.
  • Change one thing at a time and keep an audit trail stop you from losing track of what you did.
  • Check the plug and get a fresh view catch the obvious mistakes everyone overlooks.
  • If you didn't fix it, it ain't fixed closes the loop: confirm the fix against the original failure.

Deep dive: the numbers and details

This part holds the figures, equations and worked examples. Skip it if the overview is all you need. The arithmetic below is standard probability and search math, applied to the rules; it is not taken from the book.

How many runs does an intermittent bug need?

Suppose a failure appears with probability p on each independent run. The chance of seeing it at least once in n runs is 1 - (1 - p)^n. To be 95% sure of seeing it at least once:

n = ln(0.05) / ln(1 - p)
Failure rate per runRuns for 95% chance of one failure
1 in 10 (p = 0.1)29
1 in 100 (p = 0.01)299
1 in 1,000 (p = 0.001)2,995

This is why stressing the conditions matters. If a harsher setup raises the failure rate from 1 in 1,000 to 1 in 10, you need about 30 runs to catch it instead of about 3,000.

Proving the fix

The same math works in reverse for rule 9. If the bug used to fail 1 in 100 runs, then 10 clean runs prove very little: a bug that is still there would pass all 10 runs with probability 0.99^10, about 90%. To be 95% confident the failure rate has really dropped below 1 in 100, you need about 299 clean runs in a row under the same conditions. Plan the test time for this before declaring victory.

Divide and conquer in numbers

Each good/bad test that splits the search space in half removes one bit of uncertainty. Searching N candidates takes about log2(N) steps:

  • 1,000 commits since the last good build: about 10 builds and tests
  • A 64-stage data path with one wrong stage: 6 probe points, checking the middle each time
  • A 1 MB memory region with one corrupt word (262,144 words of 32 bits): 18 halving steps

Software teams automate this with git bisect run, which uses a binary search over the commit history and a script whose exit code marks each revision good or bad. Hardware teams do the same with probe points, scan dumps or simulation checkpoints.

Applying the rules in a hardware lab

RuleTypical lab practice
Make it failSweep voltage, temperature and clock; loop the failing test; capture the exact board and firmware revision
Quit thinking and lookPut a scope or logic analyzer on the real signal before theorizing; use on-chip debug such as JTAG and trace buffers
Change one thing at a timeSwap one component, one register setting or one firmware build per experiment
Keep an audit trailLog every experiment with time, setup and result, including the ones that "did nothing"
Check the plugConfirm power rails, clocks, resets, cables and the loaded firmware version first

Further reading

Share Facebook X LinkedIn WhatsApp Email

Post a Comment

0 Comments

Your comments will be moderated before it appears here.

Post a Comment (0)

#buttons=(Ok, Go it!) #days=(20)

Our website uses cookies to enhance your experience. Learn more
Ok, Go it!