test-removal-criteria

The axis: removal, not selection

Two questions about a large test suite get confused, and they have very different risk profiles:

This skill owns the second question only. Anything about narrowing a single run (impacted-set computation, previously-failing sets, fallback triggers, nightly safety runs) belongs to per-change selection and is deliberately out of scope here.

The asymmetry is the whole reason this skill is written conservatively. Meszaros names the end state of a bad removal a Lost Test, whose symptom is that "the number of tests being executed in a test suite has dropped (or has not increased as much as expected)", and lists it as a direct cause of Production Bugs: untested code has "no automated tests to tell the developer when they have introduced a problem" (Meszaros, Production Bugs). A deleted test and a disabled test produce the same silence.

How to use

1. Classify each candidate into exactly one removal class - duplicate, tautology, trivial, dead-signal, or orphan (Step 1); anything that fits no class is not a candidate.
2. Apply that class's test to confirm the classification (Step 2), honouring the never-remove list.
3. Run the four-condition delete gate; any failed or missing-input condition reverts the verdict to keep (Step 3).
4. Choose the action - keep, rewrite, fold, or remove (Step 4).
5. Record the pass as a ledger plus a kept table, packaged by confidence - templates and the confidence-to-action rules are in references/removal-ledger-and-report-format.md (Step 5).

The burden of proof sits on removal

Five rules govern every decision below. They are not advisory.

1. Keep is the default verdict. A test stays unless a removal case is affirmatively made. "No case was made either way" resolves to keep, never to delete.
2. Every removal carries a stated reason and a named reviewer. The reason names the removal class and the evidence; the reviewer is a person, not a team alias, who signed off on that specific row. A removal without both is not eligible to ship. (Practitioner convention, not a standard: teams that skip the named-reviewer column end up with rubber-stamped bulk deletes.)
3. "It never fails" is not evidence a test is worthless. A test that has passed for years while its subject churned may be the load-bearing guard that is passing because the code is still correct. Absence of failure is absence of evidence, in both directions.
4. Coverage-tool output alone is not grounds for removal. Coverage finds untested code; it does not measure test quality. Per Fowler, "if you make a certain level of coverage a target, people will try to attain it. The trouble is that high coverage numbers are too easy to reach with low quality testing" (Fowler, TestCoverage). A redundant-coverage number is one input to the delete gate below, never the whole case.
5. Never remove a test to make a signal go away. Meszaros is explicit about the temptation with an intermittently failing test: "we may be tempted to remove the failing test from the suite to 'Keep the Bar Green' but this would result in an (intentional) Lost Test" (Meszaros, Erratic Test). Flakiness is a diagnosis problem, not a removal class. Quarantine and fix.

Step 1 - Classify the candidate into one removal class

Each candidate gets exactly one class. If two fit, take the more conservative one (the one lower in this table).

A candidate that fits none of these five classes is not a removal candidate. "Old", "long", "slow", "nobody remembers why it is there", and "the coverage report says it is redundant" are not classes.

The never-remove list

Regardless of class, a test is off limits when any of these holds:

• It is labeled @critical, @regression-guard, or any team-configured do-not-delete marker.
• It has failed inside the signal window. A failure is the exact evidence that says this is a real test.
• A test-code quality review has flagged it for rewrite. Obscure or badly structured tests get refactored, not deleted: "test code is just as important as the production code and it needs to be refactored just as often" (Meszaros, Obscure Test).
• It lives in vendored or third-party paths the team does not own.
• It is currently flaky (see rule 5).

Step 2 - Apply the class test

Step 1 proposes a class from a signal; this step confirms it with a class-specific test before the gate runs. The confirmation methods are precise and easy to get wrong:

• duplicate - compare a normalized (describe-path, normalized-setup, normalized-assertion-arguments) signature, not assertion text; keep the co-located copy, and check for a fold (Step 4) first.
• tautology - confirmed when the expected side of the assertion calls into a production import (recomputes the subject), not by the literal shape expect(x).toBe(x); the default action is rewrite, not remove.
• trivial - no assertion, or a self-satisfying one; add the missing assertion if the smoke value matters, otherwise remove.
• dead-signal - requires the 90-day per-test history minimum plus a three-question reviewer checklist, reviewed one test at a time, never batched.
• orphan - confirm the imported module or symbol is genuinely gone, not moved or re-exported, then remove and name the missing module.

The full method per class - normalized signatures, detection heuristics, code examples, the dead-signal thresholds and reviewer checklist, and the supporting citations (Meszaros, Fowler, Zhang and Mesbah) - is in references/applying-the-removal-classes.md.

Step 3 - The delete gate: all four conditions must hold

Classification proposes. This gate disposes. For a removal to be eligible, every one of these must hold. If any one fails, the verdict is keep.

1. No caught regression in the window. The test has never gone from pass to fail to pass-after-fix inside the signal window. That transition is the signature of a test catching a real regression, and one occurrence is enough to end the discussion.
2. Redundant coverage, proven per test. Every source path this test covers is also covered by at least one other test, confirmed against a per-test to source-file map, not a merged coverage report. The bidirectional mapping this requires is the same one test impact analysis builds: "one test (from many) exercises a subset of the production sources" and "one prod source is exercised by a subset of the tests" (Hammant and Fowler, The Rise of Test Impact Analysis). No per-test map means condition 2 cannot be evaluated, which means no removal in that pass. Merged line coverage cannot answer "which other test covers this line".
3. No do-not-delete label. Not @critical, @regression-guard, or any team-configured equivalent.
4. Not flagged for rewrite. A test that a test-code quality review marked as obscure, weakly asserted, or badly structured is a fix, not a delete.

Two properties of this gate matter more than the conditions themselves:

• It is conjunctive and unweighted. Three strong conditions do not outvote one failing condition. Do not convert it into a score.
• A missing input is a failed condition, not a skipped one. No signal history means condition 1 fails. No per-test map means condition 2 fails. Silence never counts in favor of removal.

Step 4 - Choose the action

Removal is one of four possible outcomes, and it is the last one to reach for.

A fold looks like this:

// Before: 4 tests, 4 setup blocks.
test('addItem accepts 1',   () => { /* ... */ });
test('addItem accepts 5',   () => { /* ... */ });
test('addItem accepts 100', () => { /* ... */ });
test('addItem rejects 0',   () => { /* ... */ });

// After: 1 test, per-row failure messages preserved by the name template.
test.each([
  { qty: 1,   expected: 'accepted' },
  { qty: 5,   expected: 'accepted' },
  { qty: 100, expected: 'accepted' },
  { qty: 0,   expected: 'rejected' },
])('addItem qty=$qty is $expected', ({ qty, expected }) => { /* ... */ });

Confidence to action

Confidence is a property of the classification, not a license to skip the gate: every class still passes Step 3 before anything is removed, and classes go in separate change sets so a mix of high- and low-confidence rows never trains the reviewer to skim. The per-class confidence-to-action table - which classes may be batched, which are reviewed one at a time - is in references/removal-ledger-and-report-format.md.

Step 5 - Record and report

A removal pass produces exactly three artifacts, in the change description, before anyone approves:

1. The removal ledger - one row per proposed removal, each with a class, a reason, four gate results, and a named reviewer.
2. The kept table - every classified candidate the gate stopped, with the failing condition named. It is how a reviewer checks the gate was run rather than asserted, and the record that explains, a year later, why a quiet test was left alone.
3. A separate rewrite and fold list - not removals, and they do not ship in the same change set as deletions.

The row templates for the ledger and the kept table are in references/removal-ledger-and-report-format.md. If the pass produces test names with no gate columns and no reviewer column, it is not a removal proposal - it is a list of suspicions, and nothing on it is eligible to be deleted.

Worked example

A suite of 1,283 tests, 41 candidates surfaced, 214 days of per-test history.

Result: 3 removals across two change sets (one for duplicate plus trivial, one for orphan), 1 rewrite, 3 tests kept with recorded reasons. Candidate 5 is the case the whole gate exists for: it looked like the strongest dead-signal row in the batch until the per-test map showed it was the only test on that branch.

Anti-patterns

Limitations

• A removal decision cannot be fully automated. Classification is mechanical; the dead-signal checklist and the reviewer sign-off are not. Every rule here assumes a person reads the ledger.
• No semantic understanding of value. A test that scores as low-signal on every mechanical input may carry architectural or regression-guard value that no heuristic sees. This is why gate condition 3 depends on labels, and why unlabeled load-bearing tests are the standing risk: the team has to label them for the gate to protect them.
• Redundancy evidence is only as good as the per-test map. Tests that exercise a function indirectly through an integration path may not appear in a per-test coverage map at all, which makes redundancy look higher than it is and over-flags the dead-signal class.
• Short history means no dead-signal decisions. Under the 90-day minimum, the signal-driven classes are unusable. Duplicate, trivial, and orphan still work, because their evidence is structural rather than historical.
• Removing tests does not make a suite fast. It reduces count and maintenance surface. Per-test speed and per-change run size are different problems with different tools.