Use when changing test evidence collection, publication, paths, retention, or scratch-directory cleanup.
68%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Passed
No findings from the security scan
Fix and improve this skill with Tessl
tessl review fix ./.agents/skills/test-evidence-storage/SKILL.mdTest Evidence Stays Out of the Code Branch
StepContext.EvidenceDir; nothing stages or commits it into the pushed branch, so evidence can never reach the default branch's history. With test.evidence.store_in_repo and a derivable GitHub link base, the PR step calls publishRunEvidence (internal/pipeline/steps/evidence_publish.go), which copies the directory onto the push-target repo's orphan evidence branch through internal/evidence and hands the PR body its links. A provider without derivable links does not push the branch. Independently, test.evidence.attach_media (independent of store_in_repo) gates media uploads: on GitHub.com/GHEC it uploads image/video artifacts that pass gh's attach rules to GitHub user-attachments at PR render time (internal/scm/github/attachments.go, attachRunEvidenceMedia in evidence_attach.go), and on GitLab the same flag gates project image uploads (evidence_gitlab.go; provider defaults are owned by docs/src/content/docs/reference/global-config.md, test.evidence). GitHub uploads fail closed: upload errors keep today's rendering. One render shares one github.UserAssetSession, which runs gh auth token at most once; a failed resolution is ErrUserAssetTokenUnavailable and the step logs one line and skips the remaining uploads instead of asking the credential vault again per file (issue #1544; TestUserAssetSessionResolvesTheTokenOnce, TestUserAssetSessionTokenFailureNeverAsksAgain, TestPRStep_TokenRefusalAsksOnceAndSkipsTheRemainingFiles). Text artifacts stay inlined.internal/evidence owns the orphan-branch mechanism and its fail-closed rules: plumbing only (scratch GIT_INDEX_FILE + hash-object/write-tree/commit-tree), so HEAD, the index, and the worktree are untouched and a detached or shallow clone works; the parent is the just-fetched remote tip so the push is a plain fast-forward and never a force; an existing branch without the .no-mistakes-evidence marker at its tip is refused, which is what makes a wrong branch name (main) harmless. Valid UTF-8 text without NUL bytes passes through safepath.RedactWorktrees (the run worktree from Request.RunWorktree, then any run worktree under Request.WorktreesRoot) and then safepath.RedactText before hash-object (hashEvidenceFile; the order matters, since home redaction would stop the worktree roots matching); binary artifacts and the local files stay as written. Every failure returns an error and the PR body omits links that would not resolve; an uploaded media attachment may still render, otherwise the artifact falls back to its local-path reference.Repo.UpstreamURL/ForkURL, never the push URL, which can carry a credential.test.evidence.branch is trusted-only in EffectiveRepoConfig (it names a ref the daemon pushes to); local_root/retention/max_runs/link_base_url are global-only (applyEvidenceStorageOverrides is called from Merge with GlobalConfig alone); the rest of test.evidence (store_in_repo, attach_media, dir) stays pushed-readable. Invalid branch names, relative local_root, unparseable retention, negative max_runs, and a link_base_url that is not a bare absolute http(s) URL all fail the config at parse time (validateTestRaw). A global-only key is IGNORED from a repository config, never refused by name in parseRepoConfig: round provenance records yaml.Marshal of the effective RepoConfig and eval capture reads it back through LoadRepoFromBytes, so a by-name refusal of link_base_url broke capture for every run (TestRepoConfigProvenanceRoundTripAcceptsEvidenceLinkBaseURL).evidence_links.go (renderEvidenceFileLink): Testing local references and transcript sources (renderLocalArtifactReference), scenario Evidence cells (renderScenarioEvidenceReferences), and GitLab Evidence entries (gitlabEvidenceSection) all use it; new surfaces must too. The setting's scope, precedence, and URL layout are owned by Evidence links in docs/src/content/docs/reference/global-config.md. Regressions: TestPRStep_EvidenceLinkBaseURL_MR1016Fixture (a real MR's Testing section, unset and set), TestGitlabEvidenceLinkBaseListsImagesAsLinksWithoutUploading, TestEvidenceLinkBaseURLIsGlobalOnly.paths.ResolveEvidenceRoot and RunManager.pinRunEvidenceDir (including configured and recorded worktree overlap checks). The resolved <root>/<run-id> is pinned on runs.evidence_dir once (SetRunEvidenceDir ignores later writes); a null pin keeps legacy local_root resolution. User-facing precedence and fallback rules live in docs/src/content/docs/reference/global-config.md (evidence_dir). GitLab image upload and hash-cache rendering are owned by evidence_gitlab.go; the PR behavior contract lives in docs/src/content/docs/reference/pipeline-steps.md (Pull request).internal/evidence/publish_test.go, internal/evidence/branch_test.go, internal/pipeline/steps/evidence_publish_test.go, internal/pipeline/steps/pr_attach_media_test.go, internal/scm/github/attachments_test.go, TestPushStep_DoesNotPublishTestEvidenceIntoThePushedBranch, TestEffectiveRepoConfig_EvidenceBranchTrustedOnly, TestLoadGlobalConfig_InvalidEvidenceBranchFailsClosed, internal/config/evidence_storage_test.go.no-mistakes Owns Its Own Scratch (never the shared system temp dir)
<NM_HOME>/evidence/<runID>; custom placement follows the owners above rather than os.TempDir(). The daemon's service unit exports only HOME, PATH, and proxy vars, so TMPDIR is unset and os.TempDir() resolved to the shared /tmp - a systemd tmpfs on Ubuntu 24.10+, so evidence consumed RAM. The app root is disk-backed on all three platforms, so there is deliberately NO runtime.GOOS branch; do not add one.Executor.runEvidenceDir) into StepContext.EvidenceDir, and agent.WithSteering(a, evidenceRoot) takes it as an argument. Steps and the steering preamble must never rebuild it - two independent os.TempDir() copies is exactly the drift this replaced.RunManager.cleanupRunEvidence removes a finished run's dir when empty (os.Remove, never RemoveAll - the test step creates the dir before the agent decides it has anything to write, and that litter was 94% of observed accumulation), evidenceRootsFor discovers current and historical pinned roots, and reapLegacyEvidence adds the pre-relocation temp root before reapEvidence applies age and one shared count budget oldest-first. Removal paths reuse skipWorktreeCleanup's pending/running guard and are best effort. No OS temp timer is load-bearing.internal/eval/replay.go sandboxes stay in the system temp directory. They are the largest scratch this program creates, but a replay materializes its own nested NM_HOME and worktree while Store.Prune, the case records, and the object pools all live under <NM_HOME>/eval - so relocating the sandbox inside the app root nests it in the state it is replaying, which e2e TestEvalJourney refuses on purpose. Moving it needs a disk-backed root outside NM_HOME, which does not exist yet; do not "fix" it by weakening that assertion. Every remaining os.MkdirTemp("", ...) caller is auto-named and self-cleaning with defer; keep it that way.internal/paths/evidence_test.go, internal/config/evidence_storage_test.go, internal/daemon/evidence_reap_test.go, TestSteeringNamesTheConfiguredEvidenceRoot, TestTestEvidenceDir_DefaultResolutionStaysUnderTheAppRoot, e2e TestTestEvidenceLivesUnderAppRootNotSharedTemp / TestRunCleanupLeavesNoEmptyEvidenceDirectory.kunchenguid/no-mistakes
3d2d4bf
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.