test-design-scorecard

What this owns, and what it does not

This scorecard owns the grading layer: a 1 to 5 scale, the per-level anchors on each axis, the arithmetic that turns per-axis scores into a file score, and the feedback shape that makes a score actionable instead of demoralising.

It deliberately does not own:

Read the boundary this way: if two reviewers could disagree about what the rule is, that belongs in the conventions catalog. If they agree on the rule and disagree about how well this file follows it, that is what the anchors below exist to settle.

The scorecard never emits a verdict. A file scoring 2.1 still ships. The score is the input to a conversation and to next sprint's focus.

When to use

• Onboarding a new engineer who needs a graded, non-punitive read of their test design.
• A team deliberately ramping up test discipline and wanting a score that is comparable across files, reviewers, and sprints.
• A quarterly per-author trend where the output is a conversation, not a merge verdict.

How to use

1. Confirm this is a coaching read, not a gate. If the output must block a merge, stop and use an adversarial pass/fail review instead.
2. Score each applicable axis (Axes 1 to 6) as a whole number against its anchors; apply the 2-vs-4 tiebreaker when a file sits between two anchors.
3. Mark unmeasurable axes n/a (setup time with no timings) rather than guessing - do not score them 3 or 5.
4. Compute the file score: drop the n/a axes, average the rest unweighted, round to one decimal, and record the divisor (3.8 / 5 (5 axes)).
5. Write growth-framed feedback per the Feedback rules: strengths first, at most two growth items, each naming the current score, the target, and one concrete edit.
6. For per-PR and per-author rollups, apply the trend conventions in references/trend-and-rollup-reporting.md.

The 1 to 5 scale is a practitioner convention

The five-point scale, the anchor wording, and the numeric thresholds in the setup-time axis are conventions fixed by this scorecard so that scores are comparable across files, reviewers, and sprints. They are not drawn from any standard, and no standards body defines a "3 out of 5" test. What is sourced is the underlying design guidance each anchor encodes, cited inline per axis.

Score whole numbers per axis. Only the composite carries a decimal.

Axis 1 - AAA phase separation

Grounded in the Four-Phase Test: "Clearly identifying the four phases makes the intent of the test much easier to see" (Four-Phase Test). The three-phase naming comes from Bill Wake: "Arrange: Set up the object to be tested. Act: Act on the object (through some mutator). Assert: Make claims about the object" (Arrange-Act-Assert).

Settling 2 vs 4: count the Acts. One Act the reader can point to is a floor of 4 regardless of comment style. More than one Act is a ceiling of 2, because per the same source it "will be self-evident if we have multiple exercise SUT phases separated by result verification phases" (Four-Phase Test), and that is the split-into-two-tests signal, not a formatting nit.

Axis 2 - Single-responsibility

Grounded in the Eager Test cause of Assertion Roulette: "verifying many test conditions in a single Test Method", whose tell is that the author "wants to modify the Test Automation Framework to keep going after an assertion has failed so that the rest of the assertions can be executed" (Assertion Roulette).

Settling 2 vs 4: ask whether a failure on the first assertion hides a distinct defect. If the later assertions could have caught a different bug, they are separate conditions and the score is capped at 2. If they all fall over together, they are one condition. Bill Wake is explicit that multiple assertions are fine when they "explore a different 'dimension' of the object" (Arrange-Act-Assert), which is exactly the 4 anchor. Do not penalise three expect calls that describe one outcome.

Axis 3 - Naming

Grounded in Roy Osherove's three-part pattern [UnitOfWork_StateUnderTest_ExpectedBehavior] and his argument that when a test fails "the only thing left to save us then is the name of the test" (Naming standards for unit tests).

Settling 2 vs 4: hide the body and read the name alone. If you can state the input condition and the expected outcome, it is a 4 or 5. If you can name only the method under test, it is a 2. Count the enclosing describe chain as part of the name; a nested structure that reads as a sentence top to bottom scores the same as the flat three-part form.

Consistency cap: a suite mixing the flat pattern and the nested pattern caps at 3 on this axis even when individual names are excellent. Score the suite once for consistency, then the files individually.

Axis 4 - Fixture coupling

Grounded in the Shared Fixture trade-off: its "biggest issue" is that it "can lead to 'collisions' between tests possibly resulting in Erratic Tests since tests may depend on the outcomes of other tests", and it "will often result in a Obscure Test because the fixture is not constructed inside the test" (Shared Fixture).

Settling 2 vs 4: distinguish sharing the shape from sharing the instance. A factory that hands every test its own object shares a shape and stays at 3 or above. A fixture whose state survives from one test into the next shares an instance and caps at 2, because that is the collision path above. The test being short is not evidence of good coupling; brevity bought by an off-screen fixture is the Obscure Test the same source describes.

Axis 5 - Magic literals

Grounded in the Hard-Coded Test Data cause of Obscure Test: literals "are hard-coded in the Test Method obscuring cause-effect relationships between inputs and expected outputs", so the reader finds "it hard to determine how various hard-coded (i.e. literal) values in the test are related to each other" (Obscure Test).

Settling 2 vs 4: cover the call to the system under test and try to derive the expected value from the inputs still visible. If you can, it is a 4. If you have to run the code or open the production source to see where the number came from, it is a 2.

Do not over-apply. Naming a 1 as const QTY = 1 in a test whose whole point is that one item makes a count of one adds ceremony, not clarity. A score of 3 here is a legitimate resting place, not a defect to be fixed.

Axis 6 - Setup time

This axis requires measurement, not estimation. Instrument setup and teardown to log start and end time per test, then sort by duration: "The tests with the longest execution times are the ones where it will be most worthwhile focusing our efforts" (Slow Tests). Score from that list. If no timings exist, mark the axis n/a rather than guessing.

The 1 anchor is deliberately below the 2: reaching for a shared fixture to fix slowness trades one problem for a worse one, since "a common reaction to Slow Tests is to immediately go for a Shared Fixture but this almost always results in other problems including Erratic Tests"; the recommended move is to replace the slow component with a fake, and a fake database "can make the tests run on average fifty times faster" (Slow Tests).

Settling 2 vs 4: find out what the time is spent on. Under 1 s with no process outside the test runner is a 4. Any per-test database write, container start, or cache warm puts the ceiling at 3, and past 5 s at 2, because per the same source database-backed setup runs roughly fifty times slower than the in-memory equivalent.

Axes deliberately excluded

Assertion specificity, mocking technique, and end-to-end selector quality are not scored. Each needs a different kind of read (what the assertion would still pass for, what the double couples to, what the locator breaks on) and each is better served by a focused review than by a number. Mention them in the prose if they stand out, but keep them out of the composite so the composite stays comparable across sprints.

Computing the file score

1. Score each applicable axis as a whole number.
2. Drop axes marked n/a. Do not score them 5, and do not score them 3.
3. Average the remaining axes unweighted. Round to one decimal.
4. Record the divisor next to the score, for example 3.8 / 5 (5 axes).

Unweighted averaging is a choice: it keeps the arithmetic auditable by the person being coached. Weighting invites arguing about the weights instead of about the tests.

Feedback rules

The score is the smaller half of the output. These rules are what keep it a coaching artefact:

1. Strengths first, always at least one, with line references so it reads as observation rather than politeness.
2. Cap growth items at two per file. A long list produces no change.
3. Every growth item names the current score, the target, and one concrete edit. "Naming is a 3" is a grade. "Naming is 3 of 5; adding the scenario makes it 5, like addItem_validQty_incrementsCount" is coaching.
4. Markers: a check mark for strengths, a seedling for growth. Never a cross or a warning triangle: those are gate vocabulary and reframe the whole document as a rejection.
5. Banned words: violation, fails, wrong, must. Use consider, next time, to reach 5.
6. Score the file, never the person. No inter-author comparison anywhere in the output, including the trend report.
7. One growth focus per sprint, chosen by the author, not assigned.

Worked example: per-test feedback

### `cart.spec.ts > addItem increments count` - 4.2 / 5 (5 axes)

**Strengths:**
- ✅ AAA phase separation (5/5): arrange lines 12-14, act line 16, assert 18-20.
- ✅ Single-responsibility (5/5): one observable target, `cart.itemCount`.
- ✅ Fixture coupling (5/5): built inline, nothing imported from elsewhere.

**Growth opportunities:**
- 🌱 Naming (3/5): "addItem increments count" gives the subject and the
  outcome. Adding the scenario reaches 5:
  `addItem_validQty_incrementsCount`.
- 🌱 Magic literals (3/5): the quantity `1` and expected count `1` are
  clearly related, so this is a fine resting place. Worth naming only once
  the arithmetic stops being obvious, for example a taxed total.

**Next sprint goal:** try the three-part naming pattern on every new test in
the cart suite, then reread the suite as a group and see whether it scans.

Rollup and trend reporting

The per-test worked example above is the core artefact. Aggregating file scores into a per-PR summary and a per-author trend line - with the worked examples and the two reporting conventions (report no trend on fewer than 3 scored PRs in a period; treat a period-over-period delta under 0.3 as noise; always print the axis count) - lives in references/trend-and-rollup-reporting.md.

Anti-patterns

Limitations

• Inter-rater reliability is the real weakness. Two competent reviewers scoring the same file will often differ by a point on the judgement-heavy axes (naming, single-responsibility, fixture coupling). The anchors and the 2-vs-4 tiebreakers narrow the spread; they do not close it. Treat a one-point difference between reviewers as inside the noise floor, and keep the same scorer across a trend line wherever possible.
• Not a performance-management instrument. These numbers do not belong in a performance review, a promotion packet, or a team dashboard that compares people. The scorecard measures how legible a test file is to a reader, which is a property of the file, of the codebase's age, and of how much of it the author inherited.
• A well-formed test can still be the wrong test. Every axis scores structure. None of them checks that the assertion verifies behaviour anyone cares about, so a 5.0 file can test nothing of value.
• Context is invisible. A shared fixture may be a deliberate, documented trade-off for a suite that would otherwise take an hour. Score what you see, then ask before writing the growth item.
• Trends need consistent adoption. Scored sporadically, the line tracks which PRs happened to get reviewed, not how anyone's testing changed.