Problem: Unit-test coverage is 80% and production still breaks — what is coverage actually for? —— Trove problem library

Symptom

Start with a real case that still makes my blood run cold: a team gated every merge request with a 60% coverage red line in SonarQube and dutifully ran unit tests every time — yet P0 incidents in production went from once a quarter to three times a month.

I was called in as an agile-transformation consultant, and my first instinct was "did they actually execute the tests properly?" When I looked at the data, the picture changed: overall coverage was a beautiful 71%;

but of the ~3,500 lines added and ~1,800 lines modified per iteration, the coverage on those changed files was only 47% — and the changes that directly caused P1 incidents had a median coverage of just 23%.

The team didn't lack tests; they had written all their tests on old code that hadn't moved in two years, while the code that actually changes every day — and is most likely to break — was essentially running naked.

The more ironic way to die is another one: the coverage number keeps rising and production keeps breaking.

To hit the target, teams pour effort into patching tests for stable modules and padding coverage with getter/setter tests; developers grind the number because the KPI demands it, and testing turns from a quality guarantee into a number game.

A study of 7,000+ Java projects on GitHub showed that defect density clearly drops from 0% up to 60–70% coverage — but past that threshold, the marginal benefit of added coverage collapses, and there are even cases of "higher coverage, more defects".

And a third death I know intimately: tests become a burden and the team gives up collectively.

Tests are welded to implementation details — restructure an internal variable and everything goes red; mocks are so complex that nobody dares touch them; nobody can tell what a test is even asserting.

So everyone silently agrees: comment it out, skip it, delete it. Tests go from safety net to liability.

If any of this sounds familiar — coverage meets the bar and production still breaks; or you're frantically adding tests to hit a number and nobody maintains them; or tests have become a "comment out the red" ritual — this article is for you.

It is not a testing tutorial.

It is the consensus, bought with real incidents and data, of people who have genuinely been fooled by coverage and have also turned tests into a real safety net: coverage is a rear-view mirror, not a windshield — it tells you where you've been, not whether the road ahead is safe.

Cause

First, be clear about what coverage actually measures.

Coverage answers exactly one question: which code got executed. It cannot answer three far more important questions: ① whether the executed code was actually verified as correct (did it assert anything, or just run?);

② what risk hides in code that never executed; ③ whether change-related code was covered. Treating coverage as a quality target is like proving "this road is safe" with "I've driven it before" — the correlation is far weaker than you assume.

Why do teams collectively fall into the "higher coverage is always better" trap? Because it is the easiest quality metric to measure and the one that requires the least judgment.

Management wants quality, teams must produce numbers, and coverage is the "safety certificate" both sides can read.

But precisely because it is easy, it becomes the most dangerous self-deception tool:

First, coverage moves attention from "change risk" to "code coverage". The core activity of agile iterations is adding and modifying code, not maintaining old code.

When a team spends 80% of its testing effort on stable modules untouched for two years and leaves 20% for this iteration's new and changed code, quality collapse is only a matter of time.

Second, a coverage KPI manufactures "test theater". Once coverage becomes a performance metric, the optimal strategy is "add tests to the easiest code": getters/setters, utility functions, stable modules — these tests are fast to write and inflate coverage fast, but do nothing for real risk.

Testing turns from a quality guarantee into a number game — this is "testing becomes theater".

Third, fragile tests self-destruct. Tests welded to implementation details (internal fields, private structures) all go red on refactor; mocks get so complex that nobody dares change them; assertion-less "execution tests" run but verify nothing.

When maintenance cost exceeds benefit, teams vote with their feet: comment, skip, delete — and eventually lose even the ability to know where things broke.

And one more truth everyone knows but nobody says out loud: code without tests is code "nobody dares change", and code nobody dares change is the biggest source of technical debt. The real value of tests was never the percentage — it is the freedom to act.

Every experience below is about pulling coverage from "number game" back to "change protection".

Solution

(Consensus experience: each item is phenomenon/case → do this → why. These are repeated consensus across multiple independent practices, not one person's opinion.)

Experience 0 (L1 one-line takeaway): Coverage is a rear-view mirror — watch "change coverage", not "total coverage"

Total coverage is a rear-view mirror: it shows where you've been, not whether the road ahead is safe. What you should actually track is "is this change's code protected by tests". In a hurry?

Just do this one thing: change the quality gate from "total coverage ≥80%" to "incremental coverage ≥80% on this change", and hold tests responsible only for changed code. Experiences 1–5 below are the why and the how.

Experience 1: Test the change, not the legacy — incremental coverage is what release decisions should be based on

Phenomenon: gating releases on total coverage is a nightmare every team knows: two days before release you're 2% short, so you spend three days adding tests to code that hasn't changed in years — code that has been running fine for a long time, where the risk of touching it outweighs the benefit, and where you might even introduce new bugs.

Meanwhile, this iteration's truly new and modified high-risk code may sit at 20% coverage.

Do this: switch the gate from "total coverage" to "incremental coverage": parse the files in this change, count whether the newly added/modified lines are covered by tests, and only allow the merge when that bar is met.

One team measured it: after switching the gate from "total 85%" to "incremental 85%", developers started writing tests alongside business code, releases stopped being held hostage by legacy code, and the quality gate meant something for the first time.

Reference line: incremental coverage ≥80%, high-risk modules (money, permissions, data consistency) ≥90%.

Why: incremental coverage answers "will this change break production" — which is exactly what a release decision needs. Total coverage answers "how much accumulated test asset do we have", which has almost no predictive power for "is this release safe".

The 7,000-project GitHub evidence said it long ago: past the 60–70% threshold, the marginal value of total coverage approaches zero; while coverage of changed code almost always sits below that threshold.

Experience 2: Don't make coverage a KPI — metrics lie, tests perform

Phenomenon: once coverage enters the KPI, you get test theater: developers pick the cheapest code to cover (getters/setters, utility functions, stable modules), the coverage number climbs, and production defects don't move — or climb too.

One team ran a 60% gate for six months and P0s went from once a quarter to three times a month — not because coverage is useless, but because they treated "coverage" as "quality" itself.

Do this: coverage is a health-check metric, not a performance metric. What should you evaluate on? Outcome metrics: production defect rate, regressions introduced by changes, and the number of issues tests intercepted.

Give teams room to judge "where tests belong" instead of one blunt number that forces everyone to grind.

Turn "80% coverage" from an order into a conversation: "Which part of this change carries the most risk? Is it covered?"

Why: people optimize for what is measured. Make coverage a KPI and teams optimize the coverage number instead of quality — the classic Goodhart's law case: when a metric becomes a target, it stops being a good metric.

The real quality metric is "have tests ever stopped an incident", not "how many lines did tests run".

Experience 3: Write behavior-facing tests — tests welded to implementation details are landmines

Phenomenon: the #1 reason tests become unmaintainable: they're welded to implementation details.

The internal price list changes from a List to something else and every test goes red; mock chains are so complex that changing one parameter means touching five files;

assertion-less "execution tests" run but verify nothing, so a red light doesn't tell you whether the logic or the test is wrong.

Such tests hurt every time you touch them, and eventually the team comments them all out.

Do this: test observable behavior, not internal structure: depend only on public APIs; mock at external boundaries (external services, databases, time), never mock internal logic;

every test has at least one meaningful assertion — assert "what the system did", not "how it did it".

A "fragile test" looks like this: reaching directly into InternalPriceList's internal fields;

a "robust test" looks like this: adding items through public methods, returning prices through a fake implementation of the external service, asserting the order total — same business, and the latter costs a fraction of the former to maintain.

Why: implementation details are the part of code that changes most often. Weld tests to them and every refactor means rewriting tests — tests become a barrier to refactoring instead of an aid.

Behavior-facing tests leave "how it's implemented" as free space for refactoring: as long as external behavior doesn't change, tests stay green, and you can safely change things.

Experience 4: Don't invert the pyramid — unit tests are the base; don't make E2E carry the load

Phenomenon: many teams have an inverted test pyramid: piles of E2E tests, almost no unit tests. E2E is slow, flaky, and expensive to maintain — change a front-end button ID and dozens of cases go red at once.

And because E2E touches many code paths, it inflates your coverage number, making you think you're "well tested" when you aren't.

Do this: structure tests by pyramid ratio: a large base of fast, stable unit tests (broadest coverage), a moderate middle of integration tests (module collaboration), and a small top of key-path E2E (core user journeys).

E2E protects only "the most important things" — don't stuff everything into E2E.

Reference line: of tests added in an iteration, 80% should be unit-level; add E2E only at "core path" level.

Why: the essence of the test pyramid is "cover the most ground with the cheapest tests". Unit tests run in milliseconds, pinpoint failures, and are easy to maintain; E2E runs in minutes, is hard to pinpoint, and breaks in cascades.

Making E2E the workhorse means using the most expensive tool for the most work — the cost collapses and the whole test system falls apart.

Inverted-pyramid teams: build the unit base first; it's ten times more useful than adding more E2E.

Experience 5: In the AI era, tests are the last guardrail — AI writes code, you write tests

Phenomenon: after 2025 there's a sneakier slide: teams use AI to write code and outsource "testing" too — some even stop writing or reviewing tests because AI changes code so fast.

The result: AI-generated code merges at speed, regression protection is pure luck, and bugs slip into production quietly.

AI made "writing code" cheap — and made "not verifying" far more expensive.

Do this: testing discipline for the AI era — ①tests for AI-generated code must be written by humans (or human-reviewed): AI's tests look as correct as its code, and equally don't know what the business actually wants; ②when AI changes code, run tests before merging: make "all green" a precondition for AI merges, no matter who made the change;

③AI can help fill in tests, but humans define assertions: AI can generate skeletons and mock boundaries, but "what is correct" must be asserted by humans.

Why: AI's greatest strength is speed, and its greatest risk is also speed — errors propagate faster than ever. Tests are the only gate that can stop "AI looks right but isn't".

Outsourcing tests to AI means handing the last gate to the same fallible model that wrote the code.

Humans assert, machines verify — that is the highest-leverage division of labor in the AI era.

How to start (checklist, L1 execution layer)

1. Switch the gate to incremental: quality gate from "total coverage ≥80%" to "incremental coverage ≥80% on this change", 90% for high-risk modules;
2. Take coverage out of KPIs: evaluate on production defect rate, regressions, and tests that intercepted issues — not the coverage number;
3. Write behavior-facing tests: depend only on public APIs, mock external boundaries, at least one meaningful assertion per test;
4. Pyramid ratios: 80% of new tests unit-level; E2E only for core paths;
5. AI-code testing discipline: "all green" as a precondition for AI merges; assertions written and reviewed by humans.

How to apply this in your situation (L3 deep layer)

• Personal projects: you're both developer and tester — treat "critical-path tests" as insurance before refactoring: pin the modules you touch most with tests first (even 3–5 core cases), and you've bought the freedom to change; don't chase coverage — chase "is what I changed tested".
• Small teams (5–15 people): land incremental-coverage gating + behavior-facing tests first; that solves 80% of the "tests are useless" problem. Don't rush a coverage KPI — let tests go from "burden" to "protection" before scaling.
• Large teams: coverage red lines most easily become theater at scale — swap the KPI to outcome metrics (defect rate, regressions), use incremental coverage as the gate, use pyramid ratios to constrain structure; review quarterly "how much of our test asset actually protects changes".
• The AI era: AI writing code is already the norm; testing discipline must be stricter: tests green before AI merges, assertions human-written and human-reviewed; treat "AI writing tests" as an accelerator, not an excuse — AI can generate skeletons, but "what is correct" is always a human call.
• If you're stuck at "coverage passes and it still breaks": don't add tests first — look at two numbers first: what's the incremental coverage on this iteration's changes? What was the coverage on the changes behind the P1s? You'll most likely see: the problem isn't that there aren't enough tests — it's that the tests aren't on the code that's changing.

(Final words) If 80% coverage still ships bugs, the metric is not lying to you — you are testing the wrong places. The lesson I took from production incidents: the most valuable tests are always the ones covering the code you just changed.

Stop staring at the total coverage number for comfort.

Before you change code, check whether what you are about to touch has test coverage; if not, add three to five tests first. This matters even more now: AI writes, tests block, and writing those tests is still a human job.