CtrlK
BlogDocsLog inGet started
Tessl Logo

test-evidence-storage

Use when changing test evidence collection, publication, paths, retention, or scratch-directory cleanup.

62

Quality

78%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./.agents/skills/test-evidence-storage/SKILL.md
SKILL.md
Quality
Evals
Security

Test Evidence Stays Out of the Code Branch

  • The test step always collects evidence OUTSIDE the worktree, in the directory the executor resolved once as StepContext.EvidenceDir; nothing stages or commits it into the pushed branch, so evidence can never reach the default branch's history. With test.evidence.store_in_repo and a derivable GitHub link base, the PR step calls publishRunEvidence (internal/pipeline/steps/evidence_publish.go), which copies the directory onto the push-target repo's orphan evidence branch through internal/evidence and hands the PR body its links. A provider without derivable links does not push the branch. Independently, on GitHub.com/GHEC, test.evidence.attach_media (default true, also implied by store_in_repo) uploads image/video artifacts that pass gh's attach rules to GitHub user-attachments at PR render time (internal/scm/github/attachments.go, attachRunEvidenceMedia in evidence_attach.go). Fail closed: upload errors keep today's rendering. Text artifacts stay inlined.
  • internal/evidence owns the orphan-branch mechanism and its fail-closed rules: plumbing only (scratch GIT_INDEX_FILE + hash-object/write-tree/commit-tree), so HEAD, the index, and the worktree are untouched and a detached or shallow clone works; the parent is the just-fetched remote tip so the push is a plain fast-forward and never a force; an existing branch without the .no-mistakes-evidence marker at its tip is refused, which is what makes a wrong branch name (main) harmless. Every failure returns an error and the PR body omits links that would not resolve; an uploaded media attachment may still render, otherwise the artifact falls back to its local-path reference.
  • PR links are pinned to the evidence COMMIT, not the branch, so a later run overwriting the same paths cannot change what an old PR shows. Link bases come from Repo.UpstreamURL/ForkURL, never the push URL, which can carry a credential.
  • test.evidence.branch is trusted-only in EffectiveRepoConfig (it names a ref the daemon pushes to); local_root/retention/max_runs are global-only (applyEvidenceStorageOverrides is called from Merge with GlobalConfig alone); the rest of test.evidence (store_in_repo, attach_media, dir) stays pushed-readable. Invalid branch names, relative local_root, unparseable retention, and negative max_runs all fail the config at parse time (validateTestRaw).
  • Regressions: internal/evidence/publish_test.go, internal/evidence/branch_test.go, internal/pipeline/steps/evidence_publish_test.go, internal/pipeline/steps/pr_attach_media_test.go, internal/scm/github/attachments_test.go, TestPushStep_DoesNotPublishTestEvidenceIntoThePushedBranch, TestEffectiveRepoConfig_EvidenceBranchTrustedOnly, TestLoadGlobalConfig_InvalidEvidenceBranchFailsClosed, internal/config/evidence_storage_test.go.

no-mistakes Owns Its Own Scratch (never the shared system temp dir)

  • Evidence lives at <NM_HOME>/evidence/<runID> (paths.EvidenceDir/EvidenceRoot/RunEvidenceDir), never os.TempDir(). The daemon's service unit exports only HOME, PATH, and proxy vars, so TMPDIR is unset and os.TempDir() resolved to the shared /tmp - a systemd tmpfs on Ubuntu 24.10+, so evidence consumed RAM. The app root is disk-backed on all three platforms, so there is deliberately NO runtime.GOOS branch; do not add one.
  • One owner for the path: the executor resolves it (Executor.runEvidenceDir) into StepContext.EvidenceDir, and agent.WithSteering(a, evidenceRoot) takes it as an argument. Steps and the steering preamble must never rebuild it - two independent os.TempDir() copies is exactly the drift this replaced.
  • Cleanup is ours, in three layers: RunManager.cleanupRunEvidence removes a finished run's dir when empty (os.Remove, never RemoveAll - the test step creates the dir before the agent decides it has anything to write, and that litter was 94% of observed accumulation), reapEvidence bounds the directory by age and count oldest-first, and reapLegacyEvidence drains the pre-relocation temp directory under the same policy. All three reuse skipWorktreeCleanup's pending/running guard and are best effort. No OS temp timer is load-bearing.
  • HELD SCOPE: internal/eval/replay.go sandboxes stay in the system temp directory. They are the largest scratch this program creates, but a replay materializes its own nested NM_HOME and worktree while Store.Prune, the case records, and the object pools all live under <NM_HOME>/eval - so relocating the sandbox inside the app root nests it in the state it is replaying, which e2e TestEvalJourney refuses on purpose. Moving it needs a disk-backed root outside NM_HOME, which does not exist yet; do not "fix" it by weakening that assertion. Every remaining os.MkdirTemp("", ...) caller is auto-named and self-cleaning with defer; keep it that way.
  • Regressions: internal/paths/evidence_test.go, internal/config/evidence_storage_test.go, internal/daemon/evidence_reap_test.go, TestSteeringNamesTheConfiguredEvidenceRoot, TestTestEvidenceDir_DefaultResolutionStaysUnderTheAppRoot, e2e TestTestEvidenceLivesUnderAppRootNotSharedTemp / TestRunCleanupLeavesNoEmptyEvidenceDirectory.
Repository
kunchenguid/no-mistakes
Last updated
First committed

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.