Farley Properties and Scoring

Implementation note: The Farley Index formula and sigmoid normalization are implemented in scripts/cli_calculator.py (see calculator.md). The agent delegates all math to this deterministic Python CLI calculator (JSON in, JSON out) to ensure reproducible scores. Available commands: normalize-property, blend-scores, compute-farley, get-rating, aggregate-file, aggregate-suite, full-pipeline.

The 8 Properties of Good Tests

Source: Dave Farley, "TDD & The Properties of Good Tests"

Property interdependence: First (TDD) naturally produces Understandable and Granular tests. Atomic tests are inherently more Repeatable. Granular tests are naturally Fast. Maintainable tests remain Necessary longer. Improvement in one property often cascades; degradation often signals related degradation.

Farley Index Formula

Farley Index = (U*1.5 + M*1.5 + R*1.25 + A*1.0 + N*1.0 + G*1.0 + F*0.75 + T*1.0) / 9.0

Each property score ranges from 0.0 to 10.0. The divisor 9.0 is the sum of all weights (1.5+1.5+1.25+1.0+1.0+1.0+0.75+1.0 = 9.0), producing a final index on the 0-10 scale.

Weight Rationale

The relative ordering (U,M > R > A,N,G,T > F) is supported by cross-framework consensus. The exact magnitudes are subject to calibration against practitioner ratings.

Per-Property Scoring Rubrics (0-10 Scale)

U -- Understandable

M -- Maintainable

Tautology theatre guidance for LLM assessment: When evaluating Maintainable, the LLM must specifically check for:

• Over-specified mock interactions: Tests using verify() with exact call counts (times(1)), call ordering (InOrder), or verifyNoMoreInteractions. Ask: "Would a behaviour-preserving refactoring break this test?" If yes, the test is over-specified.
• Testing internal details via captured arguments: Tests that use ArgumentCaptor / .call_args to inspect internal state of objects passed to mocks (e.g., asserting on captor.getValue().getInternalStatus()).
• White-box mock expectations: Mock expectations that mirror internal if/else branches -- e.g., verify(service).getPremiumDiscount() paired with verify(service, never()).getStandardDiscount().
• High verify-to-assert ratio: Tests with many verify() calls but few assertEquals() calls test interactions (implementation) rather than outcomes (behaviour).
• Note: Simple verify(mock).method() without exact counts is legitimate for verifying side effects. Only flag when verification over-constrains implementation.

R -- Repeatable

A -- Atomic

N -- Necessary

Tautology theatre guidance for LLM assessment: When evaluating Necessary, the LLM must specifically check for all four types of tautology theatre -- tests whose outcome is predetermined, independent of production code:

• Mock tautology: Tests that configure a mock's return value and then assert that the mock returns that same value, with no production code in between. Logically equivalent to x = 5; assert x == 5.
• Mock-only test: Tests where every object is a mock and no real class is instantiated. Ask: "If I deleted all production code, would this test still pass?" If yes, the test has zero value.
• Trivial tautology: Assertions that are always true regardless of any code: assertTrue(true), assertEquals(1, 1), assertNotNull(new Object()).
• Framework test: Tests that verify language or framework behavior, not application code: assertNotNull(mock(Foo.class)), assertTrue("hello".contains("ell")).
• General test: Any test whose outcome is predetermined by its own setup, independent of production code behaviour.

G -- Granular

Note: "single outcome" is distinct from "single assertion statement." Multiple assertions verifying one outcome (e.g., status code and body of an HTTP response) are acceptable. Detect multiple unrelated assertions, not logical groups.

F -- Fast

T -- First (TDD Evidence)

Tautology theatre guidance for LLM assessment: When evaluating First (TDD), the LLM should note that:

• Tests with no production code exercised (all mocks, no real SUT) could never have been written test-first, since there was nothing to drive the design of.
• Mock-heavy tests that only verify interactions suggest the developer wrote the production code first and then wrote tests to confirm what the code already does ("test-after verification"), not to drive its design.
• TDD-driven tests naturally focus on observable outputs because the test is written before the implementation details exist.

Two-Phase Assessment

Phase 1: Static Analysis (deterministic)

• Parse test files, count signals per property (see signal-detection-patterns.md)
• Compute raw metrics: assertion counts, sleep presence, reflection usage, naming patterns
• Normalize metrics to per-property sub-scores using sigmoid normalization
• Produce a "static floor" score that is deterministic and reproducible

Phase 2: LLM Assessment (controlled non-determinism)

• Read tests holistically: does the test suite feel understandable?
• Assess naming quality semantically (beyond pattern matching)
• Detect tautology theatre: identify all four types (mock tautologies, mock-only tests, trivial tautologies, framework tests) plus over-specified mock interactions and tests coupled to implementation details rather than behaviour (see per-property "Tautology theatre guidance" sections above)
• Evaluate TDD evidence from design patterns static analysis cannot detect
• Adjust static scores up or down based on contextual judgment
• Produce a "judgment delta" for each property

Blending

final_property_score = 0.60 * static_score + 0.40 * llm_score

Per property, then aggregated via the Farley Index formula. The 60/40 split prioritizes deterministic measurement while incorporating semantic judgment.

LLM Reproducibility Protocol

• Evaluate all test files if under 50 files; SHA-256 deterministic selection (30%) for larger suites
• Structured rubric: evaluate each property on the 0-10 rubric above, providing specific code evidence
• Per-file evaluation is optional: when used, score each property per file and aggregate with the calculator's aggregate-suite (LOC-weighted); otherwise count signals across the whole analysed suite and run full-pipeline once
• Record the model identifier in the report for reproducibility tracking

Sigmoid Normalization

Each property uses sigmoid normalization to map raw signal counts to the 0-10 scale:

raw_negative_density = negative_signal_count / total_test_methods
raw_positive_density = positive_signal_count / total_test_methods

negative_component = (1 - sigmoid(raw_negative_density, midpoint_neg, steepness_neg)) * 10.0
positive_component = sigmoid(raw_positive_density, midpoint_pos, steepness_pos) * 10.0

property_score = neg_weight * negative_component + pos_weight * positive_component

Base score when no signals detected: 5.0 (conservative -- no-signal yields "Fair" rating, not "Good").

Rating Scale

Aggregation Levels

Scores are produced at three levels:

1. Per-test-method: Individual test quality assessment
2. Per-test-file: Aggregated across all test methods in a file (mean for positive signals, P90 for negative signals -- worst offenders must surface)
3. Per-test-suite: Aggregated across all test files in scope (LOC-weighted mean across files)

Properties Not Captured

Beck's Test Desiderata includes properties outside the Farley framework that are noted in reports as "dimensions not measured":