Break your design on purpose: what mutation testing shows that passing tests cannot

Imagine you test a smoke alarm by holding a lit match under it. If it screams, good. If it stays silent, you just learned something that no amount of staring at a green light would have told you.

Hardware verification has the same problem. You write tests and proofs, they all pass, and the dashboard is green. But a green dashboard has two very different causes: the design is right, or the checks are too weak to notice anything. From the outside they look identical.

Mutation testing is the match under the smoke alarm. This post explains it from scratch, shows what it found on a real chip project (including a regression that passed 20 out of 20 while four planted bugs walked straight through), and gives you a method you can apply to your own tests this week.

The problem: a passing test is not evidence of a good test

Think about how you decide a test suite is “good enough”. Usually it is one of:

• “It passes.”
• “Line coverage is 95%.”
• “We have 200 tests.”

None of these answers the real question: if there were a bug here, would anything fail?

Coverage says a line was executed, not that anything checked its result. A test can run a line of code and never look at what it produced. It is like a night guard who walks past every door and never tries a handle.

Key Lesson: Passing tells you the design survived your checks. It tells you nothing about how sharp the checks are. You need a second experiment to measure that.

What mutation testing is

The method has five steps:

1. Start with a design whose checks all pass.
2. Make one small, deliberate mistake in it. This broken copy is a mutant. For example, change a + to a -, flip a condition, or connect a signal to the wrong place.
3. Run all your checks against the mutant.
4. If any check fails, the mutant is killed. Your checks noticed.
5. If every check still passes, the mutant survived. You have found a blind spot.

Then repeat with another mistake. At the end you have a score: how many of the mistakes did your checks catch?

        working design
              |
     +--------+--------+--------+
     |        |        |        |
  mutant 1  mutant 2  mutant 3  ...   (one small mistake each)
     |        |        |        |
  run all your checks on each
     |        |        |        |
  KILLED   KILLED   SURVIVED          <- survivors show where
                                         checks are missing

Notice what is being measured. It is not the design. The design is fine. You are grading your checks.

Result	What it means	What you do
Killed	A check noticed the mistake	Nothing, the check earns its place
Survived	No check noticed	Find out why (see below), usually add a check

Why this finds problems that nothing else does

Here is a real example. Our project has a transmit path with a regression of 20 test jobs: functional tests, key tests, error tests, random traffic. All 20 passed, every time.

We planted single-line mistakes in one block (a unit that checks stored data for corruption using parity bits) and re-ran the same 20 jobs on them.

Of the mistakes we compared, four were not detected. 20 out of 20 still passed. (A full disk stopped us before we compared five more.)

For example, one mutant made the corruption check read the wrong slot. The tests never corrupted anything on purpose, so the checker was never forced to prove it could notice. It was a smoke alarm nobody had ever lit a match under.

The fix was to write tests that deliberately corrupt stored data, one read point at a time, and check that the chip reacts: raises a fault, stops the affected packet, refuses further traffic. That is 22 new tests. We then planted 25 single-line mistakes and ran the new tests against each.

Stage	Planted mistakes	Caught
Existing 20-job regression (compared on 4 mistakes)	4	0
New fault-injection tests	25	19

19 of 25 caught. What about the other 6? That brings us to the most important skill in mutation testing.

Key Lesson: A regression that always passes may be a regression that cannot fail. Mutation testing is how you find out which one you have.

A survivor is a question, not a verdict

When a mutant survives there are three possible reasons, and telling them apart is the real engineering work.

Why it survived	Example from our six	What to do
The mistake is harmless (an “equivalent mutant”)	Widening an enable signal that another signal already gates; the output cannot differ	Nothing. Record why it is equivalent
The mistake is real but needs a rare situation	A fault only matters if two faults land in the same cycle while a slot just lost eligibility	Write a targeted test, or accept the risk and write it down
The checks have a genuine hole	The check never looked at that signal	Add the check

Of our six survivors, three were equivalent (the design already ruled the difference out), and three needed a rare situation, such as two different faults landing in the same cycle, that we chose to note rather than chase. None turned out to be a plain hole in the checks, which is a good sign, but we only know it because we looked at each one.

An important warning: do not chase a 100% score. Equivalent mutants make it impossible, and forcing a test for a harmless mistake just adds noise. The score is a guide to where to look, not a grade.

Key Lesson: Every survivor deserves an explanation in writing: harmless, rare, or a hole. The explanation is the valuable output, more than the score.

Mutation with formal proofs: the size trap

The same idea works on formal proofs, tools that search all possible behaviours of a small design for a rule violation. There the lesson is slightly different.

We rewrote a small allocator (it hands out numbered tickets from a pool, like a deli counter) to be much faster. Then we planted nine mistakes, such as “the tree picks the wrong bit” and “freeing a ticket sets the wrong bit”, and ran the proof on nine different pool configurations.

Mistake	Caught on how many of the 9 configurations
tree stops combining too early	2
unused padding counted as free tickets	4
free-ticket counter adds the wrong thing	8
idle machine shows a non-zero ticket	8
five others	9

All nine were killed, but look at the first two rows. One mistake was only visible in 2 of 9 configurations. If we had run the proof on a single configuration, there was a real chance of choosing one that misses it, and we would have called the proof strong.

Key Lesson: Some bugs only show at certain sizes or settings. Mutation testing across several configurations is how you discover which ones.

A caution: your harness can lie too

Mutation testing needs a script that plants mistakes and counts results, and the script is software, so it can be wrong. Ours was. The proof tool found a planted bug instantly, then crashed while drawing the reproduction picture, and the script read the crash as “the bug survived”. The opposite of the truth.

The habit that prevents it is small: treat found, missed and tool broke as three separate outcomes, and never let a broken run look like success. Also give every tool call a time limit, because a frozen run is a result too.

How to try this on your own project this week

1. Pick one block that matters and whose checks you trust.
2. Plant five to ten one-line mistakes. Good ones: swap two signals, change < to <=, drop a condition, connect to the wrong index, forget to reset a register.
3. Run your whole regression on each, one at a time.
4. For every survivor, write down which of the three reasons applies.
5. For each real hole, add a test or a check, re-run the mutant, and confirm it is now killed.

Do not automate everything first. Doing it by hand teaches you more about your checks in one afternoon than any metric.

What this does not prove

• Mistakes you did not think to plant stay invisible. Mutation testing measures your checks against the mistakes you tried, not against all possible bugs.
• Our 25 planted mistakes and 9 allocator mistakes are small samples chosen by hand.
• The allocator proof covers the sizes in its sweep, up to 33 tickets, not every size. We label that block Tier B, short of the strongest “formally verified” claim.
• We have not shown the 6 survivors cannot matter, only that we understand why each survived.

Lessons

1. A passing test shows the design survived the checks. Only a broken design shows how good the checks are.
2. Break one thing at a time and count how many breaks you catch.
3. Treat every survivor as a question and write down the answer.
4. Try several configurations. Bugs hide in the ones you did not run.
5. Check the checker: a broken harness can report the opposite of the truth.

Where to go next

• A passing testbench proves nothing
• What a formal pass has to earn
• The bug formal verification never saw