Testing Standards

Scope

• Governs code the project ships: the published artifact, everything CI builds from it, and the tests covering both
• Narrow exception for an instrument tree.
• Instrument tree: code whose OUTPUT is the deliverable, read by a human (live-upstream probes, reverse-engineering scratch, one-off measurement scripts)
• An instrument tree falls outside this rule in full — every section below, not Coverage alone
• Preconditions (all required):
  1. The project's publish-exclude manifest lists the tree (.tesslignore, .npmignore, MANIFEST.in) — readable from the repo under review, never asserted in prose
  2. No shipped module imports from the tree
  3. Deterministic logic the tree's findings rest on carries tests, wherever those tests live
• A module the published artifact reaches does NOT qualify, whatever directory holds it
• "It is scratch" does NOT qualify on its own — all three preconditions are required
• Every other tree in the repo follows this rule in full

Coverage

• Every module gets tests — no untested code ships (narrow exception: Platform-Bound Untestable Carve-Out below)
• Test file naming follows the project's convention, or the Stack Conventions table when the project has none
• Narrow exception for code that cannot execute on the project's CI runners.
• Applies when the code drives an external desktop app, OS-level automation, or a proprietary runtime the runners cannot host (GUI automation gated on OS consent, a device-only hardware API)
• Preconditions (all required):
  1. The deterministic, CI-runnable pieces are extracted and unit-tested — parsing, normalization, and data-shaping helpers split out of the automation wrapper. Only the genuinely-unhostable layer is exempt
  2. A manual validation procedure for the exempt layer is documented — what to run, what to observe, what counts as a pass
  3. The project's README or contributor doc names this carve-out, each exempt artifact, and where its validation procedure lives
• "Hard to install in CI" does NOT qualify — install the tool and test it per rules/ci-safety.md Install, Don't Skip
• Every other module still ships tests that run in CI
• A guard is a shipped or CI check that decides pass or fail: a validator, a lint, a gate, a hook, a preflight, a verdict parser
• A test's own assertions are not guards
• A new or changed guard ships with a test feeding it one input it must reject and one input it must accept
• Use real historical artifacts as those inputs where they exist
• A guard tested in one direction alone is untested
• The accept case runs through the guard's documented entry point: the hook as the agent invokes it, the CLI, the workflow
• A config change is tested in every environment class it reaches

Stack Conventions

• An existing project's convention wins over this table

Assertions

• Assert outcomes, not implementation details
• Test what the code does, not how it does it
• If an internal refactor breaks your tests, the tests were testing the wrong thing
• Mock at the process boundary — the network, the clock, the filesystem, a paid external service — never the code under test's own collaborators
• A test whose mocks outnumber its assertions tests the mocks
• A boundary mock proves shape, never validity — text crossing a boundary (SQL, an API request) runs once against the real counterparty or a contract recorded from it
• A status write ships with a test where its dependency fails and the recorded status is failed or unknown
• A test that never runs the code under test, or observes none of its outcomes, is documentation, never a test
• A raised error, a crash, or a timeout is an observed outcome
• Assert a value that encodes a product decision as a literal, never as a value recomputed by the code under test
• A test that greps the code under test's own source guards a name, never behavior — exercise the code instead
• A run that executes zero tests is a failure — check the reported test count, never the exit code alone
• Narrow exception for mocking an in-process collaborator.
• Preconditions (all required):
  1. The user stated permission for that collaborator in the current task
  2. The test names the collaborator it mocks and the outcome the mock stands in for
• Every other in-process collaborator is exercised for real

Determinism

• Tests must be deterministic — no self-generated random test data (narrow exception: seeded property-based tests below)
• Provide fixed test data; never let runtime randomness generate or shape the test's inputs
• No dependence on the current date or wall-clock time — no today, runtime now(), or hardcoded future dates in assertions or fixtures
• Control the clock: inject or freeze "now" (a passed-in reference date, a mocked time source, a Clock instance) so a test green today is green every day
• Fixed past dates as fixtures are fine — the ban is on time-relative values that rot as the run date advances
• Flaky tests are bugs — diagnose the root cause, don't retry and hope
• A non-deterministic evaluation (an eval, a model or tier trial) runs at least 3 times and reports the spread, never the best run
• Narrow exception for property-based tests whose generated inputs come from a pinned seed.
• Applies when a generative testing library (Kotest property tests, Hypothesis, fast-check, SwiftCheck) explores many cases per run under a fixed seed
• Preconditions (all required):
  1. The generator seed is a constant in the test file (e.g. PropTestConfig(seed = 1234)), never drawn from the clock, the environment, or an unset default
  2. The iteration count is explicitly bounded (e.g. iterations = 100), never unbounded or environment-derived
  3. No runtime-RNG call (random(), unseeded Random(), shuffle) selects or shapes the inputs
• Every other case still follows Determinism: unseeded or unbounded generation, and any runtime-RNG input, stays forbidden
• Prefer a deterministic outcome (process dead, exit code) over an elapsed-time check
• Narrow exception for elapsed-time bounds in tests of code that enforces a deadline.
• Preconditions (all required):
  1. The code under test enforces a timeout or deadline, and the test exists to prove it stops waiting
  2. The test asserts only an upper bound on elapsed time — never an exact duration, a lower bound, or ordering by elapsed time
  3. The bound is a named constant in the test, at least 4× the deadline under test plus fixed slack
  4. No date, time-of-day, or runtime now() value enters the test's inputs or assertions
• Every other elapsed-time assertion stays forbidden

Fixtures and Independence

• No binary fixtures checked into the repo
• Build test data programmatically in test setup/fixtures
• For binary files that can't be built programmatically, download them from a URL during test setup
• Each test must run independently — no shared mutable state between tests
• Test order must not matter
• Tests never read machine-local paths (a home directory, a user's reports directory)
• Clean up after yourself: temporary files, database state, mock patches