The QA Metrics That Actually Matter (and the Ones That Don't)
The Vanity Metric Trap
"We have 2,400 automated tests with a 98% pass rate" sounds impressive in a status update. It is also almost meaningless. A pass rate tells you nothing about whether those 2,400 tests cover the parts of your product that actually matter, and a suite can maintain a 98% pass rate for months while critical bugs slip past it entirely. The metrics below are the ones that correlate with actual product quality — not the ones that just look good in a slide deck.
1. Defect Escape Rate
What it measures: The percentage of bugs found by customers in production, versus bugs caught internally before release.
Formula: Escape Rate = Bugs Found in Production / (Bugs Found in Production + Bugs Found in QA) × 100
Why it matters more than pass rate: A test suite can pass 100% of the time and still miss entire categories of bugs it was never written to catch. Escape rate measures the thing you actually care about — quality as experienced by the customer — not the health of your test suite in isolation.
Healthy target: Under 10-15% for most SaaS products; mission-critical systems (healthcare, finance) should aim lower.
2. Mean Time to Detect (MTTD)
What it measures: The average time between a defect being introduced (i.e., the commit that caused it) and the team becoming aware of it.
Why it matters: A bug caught in code review costs minutes to fix. The same bug caught by a customer three weeks later costs hours of investigation just to reproduce, plus the reputational cost. MTTD is a direct proxy for how early your quality gates are catching problems.
How to improve it: Faster CI feedback loops, better production monitoring/alerting, and shifting more testing left into code review and pre-merge checks.
3. Mean Time to Resolve (MTTR)
What it measures: Once a defect is confirmed, how long until a fix is deployed to production.
Why it matters: MTTD tells you how fast you notice; MTTR tells you how fast you react. A team with fast detection but slow resolution still leaves customers exposed to bugs for a long time. Track this by severity — a Critical bug's MTTR (hours) should look very different from a Low severity bug's MTTR (weeks).
4. Requirements/Risk Traceability
What it measures: The percentage of defined requirements or identified risk areas that have at least one corresponding test case.
Why it matters: This is the metric that catches the "we have lots of tests but they're not testing the right things" problem. A team can have thousands of tests and 40% traceability if all those tests cluster around a handful of easy, well-understood features while high-risk areas (billing, permissions, data export) go untested.
How to calculate it: Maintain a simple mapping in your test management tool — every requirement or identified risk gets linked to the test case(s) that cover it. Anything unlinked is a visible gap.
5. Automation Flakiness Rate
What it measures: The percentage of automated test runs that fail intermittently without any underlying code change.
Formula: Flakiness Rate = Flaky Failures / Total Test Runs × 100
Why it matters: Flaky tests are corrosive to engineering culture. Once developers see a test fail and then pass on re-run a few times, they stop trusting all failures — including real ones — and start reflexively re-running the pipeline instead of investigating. A flakiness rate above 5% is usually the point where trust collapses.
How to manage it: Track flakiness per-test, not just suite-wide. Quarantine the worst offenders into a non-blocking job, and treat fixing them as a first-class backlog item, not an afterthought.
The Metrics Worth Retiring
- Raw test case count: "We have 5,000 test cases" says nothing about coverage quality. A team can have 5,000 shallow tests or 500 deep, well-targeted ones — the second is more valuable.
- Pass rate alone, without context: As discussed above, a high pass rate on a suite that doesn't cover the right things is a false signal of health.
- Bugs found per release, without severity weighting: Ten cosmetic bugs found is not equivalent to one data-loss bug found. Weight by severity before comparing releases.
Building a Dashboard That Drives Decisions
Track these five metrics on a rolling monthly basis, and — critically — review the trend, not the single data point. A defect escape rate of 12% this month is only meaningful in the context of whether it was 8% or 20% last month. Bring this dashboard to leadership reviews, and use it to justify where QA investment goes next: if MTTD is climbing, invest in monitoring; if flakiness is climbing, invest in test suite health; if traceability on a risk area is low, invest in coverage there specifically.