regression-suite-selector

Overview

Test Impact Analysis (TIA) runs only the tests a change can affect: map production sources to the tests that exercise them, then intersect that map with the PR diff (tia-fowler). The map is bidirectional - one test exercises a subset of sources; one source is exercised by a subset of tests. Azure Pipelines builds it from per-test dynamic dependencies; Bazel derives it statically from the build graph (tia-azure).

This skill builds a TIA-style selector for any team - stitching coverage data, git diff, and a safe fallback policy - without vendor tooling.

When to use

• The regression suite takes >10 min on a PR; CI is the development bottleneck.
• A nightly / pre-release full run exists and can serve as the safety net for selection misses.
• The team has coverage instrumentation (lcov-analysis, etc., in the qa-test-reporting plugin, or js-unit-tests in qa-unit-tests-js) - the per-test → source map is computable from it.

If the build is a Bazel / Pants / Buck monorepo, the selection already comes from the build graph (Step 5) and this skill is mostly orchestration around it.

Step 1 - Decide the selection policy

A robust selector runs impacted + previously-failing + newly-added tests, and falls back to the full suite for any change it can't map (tia-azure).

The selection set per PR:

impacted ∪ previously_failing ∪ newly_added ∪ (FALLBACK if any change is unmappable)

Hard-coded fallback triggers (run everything):

• Build config changes (pom.xml, package.json, Cargo.toml, requirements.txt, Dockerfile).
• CI workflow changes.
• Files outside the source-coverage map (a new asset type the map doesn't know about).
• N PRs since the last full run (configurable; default: every 5 PRs to a branch).

Azure's TIA reasons only about managed code on a single machine; it can't reason about HTML / CSS changes and falls back to running all tests (tia-azure).

Step 2 - Build the per-test → source map

Emit per-test coverage (not merged), then invert it into a file → tests map. The core inversion:

# scripts/build_test_map.py
def build_map(per_test_coverage):
    """returns {file_path: [test_id, ...]}"""
    inverted = defaultdict(list)
    for test_id, coverage in per_test_coverage.items():
        for file_path, hits in coverage.items():
            if any(h > 0 for h in hits):
                inverted[file_path].append(test_id)
    return dict(inverted)

Persist as test-map.json, checked into the repo or stored as a CI artifact updated on every main run. Per-framework recipes for emitting per-test coverage (Jest, pytest + coverage.py, JaCoCo) and the Bazel / Pants / Buck build-graph path (where rdeps gives the map directly) live in references/instrumentation-and-ci.md.

Step 3 - Compute the changed-file set

git diff --name-only origin/${{ github.base_ref }}...HEAD

Important: ... (three dots), not ... Three-dot diff is "what changed on this branch since it diverged from main", which matches PR semantics. Two-dot diff is "differences vs current main HEAD" which can show changes the PR didn't make if main moved forward.

Step 4 - Combine

def select_tests(map, changed_files, previously_failing, newly_added):
    impacted = set()
    for f in changed_files:
        if f in map:
            impacted.update(map[f])
        else:
            return ('FALLBACK', f)   # unknown file type
    return ('SELECTED', impacted | previously_failing | newly_added)

previously_failing comes from the most recent full-suite run on main (CI artifact). newly_added comes from git diff --diff-filter=A --name-only filtered to test files.

Step 5 - Add safety: periodic full run + drift detection

Pair selection with a full-suite run so a stale map can't silently shrink coverage. Azure's pattern: run selected tests (T1) and all tests (T2) in sequence and confirm T2 reports the same result as T1 (tia-azure).

• Pattern A - Nightly full-suite run. Cron a full-suite job nightly. Failures that didn't appear in PR runs reveal selection misses; investigate and update the map.
• Pattern B - N-th PR full run. Every N-th PR (e.g. every 5th, configurable) runs the full suite as a "shadow" - silent if it agrees with selection, a warning issue if it doesn't.

CI workflow (selected run + shadow full run): references/instrumentation-and-ci.md.

Step 6 - Surface the selection

PR-comment summary so reviewers know what ran:

## Test Impact Analysis - `<sha>`

**Selected:** 47 tests of 1,283 total (3.7%)
**Strategy:** impacted ∪ previously_failing ∪ newly_added
**Reason for selection:**

| Source              | Tests added |
|---------------------|------------:|
| Impacted by changes |          39 |
| Previously failing   |           5 |
| Newly added          |           3 |

**Files driving impacted set:**
- `src/checkout/cart.ts` → 12 tests
- `src/checkout/promo.ts` → 18 tests
- `src/api/orders.ts` → 9 tests

**Last full-suite run:** 2026-05-04 22:00 UTC (12 hours ago) - passed.

Step 7 - Configurable overrides

The team should be able to opt out for a specific build, matching Azure's DisableTestImpactAnalysis build-variable escape hatch (tia-azure):

• PR label run-all-tests → forces full suite for that PR.
• Path filter tia-include → only consider TIA for changes matching this pattern (matches Azure's TIA_IncludePathFilters per tia-azure).
• PR title [full-suite] → forces full suite.

Anti-patterns

Limitations

• Map quality bounds selection quality. A map missing edges = missed tests. Update on every main full run.
• Build-config / CI-config / dependency changes are unmappable. The fallback rule (Step 1) is non-negotiable.
• Per-test coverage instrumentation has overhead. ~30 - 60% slower than merged coverage. Run on main only; PRs use the artifact.
• Multi-machine topologies break the mapping. Per tia-azure, Microsoft's TIA is "single machine topology" only - the same applies here unless the test/SUT topology is captured in the map.
• Doesn't fix slow tests. Selection cuts the count; per-test speed is still on the team. Pair with test-coverage-targeter for the "what to add next" side.

References

• tia-fowler - Hammant + Fowler on Test Impact Analysis: bidirectional mapping, Microsoft's investment since 2009, Google Blaze approach.
• tia-azure - Azure Pipelines TIA: selection mechanism, safe fallback, configurable overrides, what's NOT supported (data driven tests, multi-machine, .NET Core, UWP), the "Run TIA + run all in sequence" comparison pattern.
• bazel-deps - Bazel target dependency model, rdeps reverse dependency query, declared-vs-actual dependency principle.
• test-coverage-targeter (qa-test-reporting) - its coverage debt ledger tracks files that lost coverage / went stale.