CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

test-evidence-storage

Use when changing test evidence collection, publication, paths, retention, or scratch-directory cleanup.

Quality

68%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./.agents/skills/test-evidence-storage/SKILL.md
SKILL.md
Quality
Evals
Security

Test Evidence Stays Out of the Code Branch

  • The test step always collects evidence OUTSIDE the worktree, in the directory the executor resolved once as StepContext.EvidenceDir; nothing stages or commits it into the pushed branch, so evidence can never reach the default branch's history. With test.evidence.store_in_repo and a derivable GitHub link base, the PR step calls publishRunEvidence (internal/pipeline/steps/evidence_publish.go), which copies the directory onto the push-target repo's orphan evidence branch through internal/evidence and hands the PR body its links. A provider without derivable links does not push the branch. Independently, test.evidence.attach_media (independent of store_in_repo) gates media uploads: on GitHub.com/GHEC it uploads image/video artifacts that pass gh's attach rules to GitHub user-attachments at PR render time (internal/scm/github/attachments.go, attachRunEvidenceMedia in evidence_attach.go), and on GitLab the same flag gates project image uploads (evidence_gitlab.go; provider defaults are owned by docs/src/content/docs/reference/global-config.md, test.evidence). GitHub uploads fail closed: upload errors keep today's rendering. One render shares one github.UserAssetSession, which runs gh auth token at most once; a failed resolution is ErrUserAssetTokenUnavailable and the step logs one line and skips the remaining uploads instead of asking the credential vault again per file (issue #1544; TestUserAssetSessionResolvesTheTokenOnce, TestUserAssetSessionTokenFailureNeverAsksAgain, TestPRStep_TokenRefusalAsksOnceAndSkipsTheRemainingFiles). Text artifacts stay inlined.
  • internal/evidence owns the orphan-branch mechanism and its fail-closed rules: plumbing only (scratch GIT_INDEX_FILE + hash-object/write-tree/commit-tree), so HEAD, the index, and the worktree are untouched and a detached or shallow clone works; the parent is the just-fetched remote tip so the push is a plain fast-forward and never a force; an existing branch without the .no-mistakes-evidence marker at its tip is refused, which is what makes a wrong branch name (main) harmless. Valid UTF-8 text without NUL bytes passes through safepath.RedactWorktrees (the run worktree from Request.RunWorktree, then any run worktree under Request.WorktreesRoot) and then safepath.RedactText before hash-object (hashEvidenceFile; the order matters, since home redaction would stop the worktree roots matching); binary artifacts and the local files stay as written. Every failure returns an error and the PR body omits links that would not resolve; an uploaded media attachment may still render, otherwise the artifact falls back to its local-path reference.
  • PR links are pinned to the evidence COMMIT, not the branch, so a later run overwriting the same paths cannot change what an old PR shows. Link bases come from Repo.UpstreamURL/ForkURL, never the push URL, which can carry a credential.
  • test.evidence.branch is trusted-only in EffectiveRepoConfig (it names a ref the daemon pushes to); local_root/retention/max_runs/link_base_url are global-only (applyEvidenceStorageOverrides is called from Merge with GlobalConfig alone); the rest of test.evidence (store_in_repo, attach_media, dir) stays pushed-readable. Invalid branch names, relative local_root, unparseable retention, negative max_runs, and a link_base_url that is not a bare absolute http(s) URL all fail the config at parse time (validateTestRaw). A global-only key is IGNORED from a repository config, never refused by name in parseRepoConfig: round provenance records yaml.Marshal of the effective RepoConfig and eval capture reads it back through LoadRepoFromBytes, so a by-name refusal of link_base_url broke capture for every run (TestRepoConfigProvenanceRoundTripAcceptsEvidenceLinkBaseURL).
  • Evidence-link rendering is owned by evidence_links.go (renderEvidenceFileLink): Testing local references and transcript sources (renderLocalArtifactReference), scenario Evidence cells (renderScenarioEvidenceReferences), and GitLab Evidence entries (gitlabEvidenceSection) all use it; new surfaces must too. The setting's scope, precedence, and URL layout are owned by Evidence links in docs/src/content/docs/reference/global-config.md. Regressions: TestPRStep_EvidenceLinkBaseURL_MR1016Fixture (a real MR's Testing section, unset and set), TestGitlabEvidenceLinkBaseListsImagesAsLinksWithoutUploading, TestEvidenceLinkBaseURLIsGlobalOnly.
  • Evidence placement is owned by paths.ResolveEvidenceRoot and RunManager.pinRunEvidenceDir (including configured and recorded worktree overlap checks). The resolved <root>/<run-id> is pinned on runs.evidence_dir once (SetRunEvidenceDir ignores later writes); a null pin keeps legacy local_root resolution. User-facing precedence and fallback rules live in docs/src/content/docs/reference/global-config.md (evidence_dir). GitLab image upload and hash-cache rendering are owned by evidence_gitlab.go; the PR behavior contract lives in docs/src/content/docs/reference/pipeline-steps.md (Pull request).
  • Regressions: internal/evidence/publish_test.go, internal/evidence/branch_test.go, internal/pipeline/steps/evidence_publish_test.go, internal/pipeline/steps/pr_attach_media_test.go, internal/scm/github/attachments_test.go, TestPushStep_DoesNotPublishTestEvidenceIntoThePushedBranch, TestEffectiveRepoConfig_EvidenceBranchTrustedOnly, TestLoadGlobalConfig_InvalidEvidenceBranchFailsClosed, internal/config/evidence_storage_test.go.

no-mistakes Owns Its Own Scratch (never the shared system temp dir)

  • Evidence defaults to <NM_HOME>/evidence/<runID>; custom placement follows the owners above rather than os.TempDir(). The daemon's service unit exports only HOME, PATH, and proxy vars, so TMPDIR is unset and os.TempDir() resolved to the shared /tmp - a systemd tmpfs on Ubuntu 24.10+, so evidence consumed RAM. The app root is disk-backed on all three platforms, so there is deliberately NO runtime.GOOS branch; do not add one.
  • One owner for the path: the executor resolves it (Executor.runEvidenceDir) into StepContext.EvidenceDir, and agent.WithSteering(a, evidenceRoot) takes it as an argument. Steps and the steering preamble must never rebuild it - two independent os.TempDir() copies is exactly the drift this replaced.
  • Cleanup is ours: RunManager.cleanupRunEvidence removes a finished run's dir when empty (os.Remove, never RemoveAll - the test step creates the dir before the agent decides it has anything to write, and that litter was 94% of observed accumulation), evidenceRootsFor discovers current and historical pinned roots, and reapLegacyEvidence adds the pre-relocation temp root before reapEvidence applies age and one shared count budget oldest-first. Removal paths reuse skipWorktreeCleanup's pending/running guard and are best effort. No OS temp timer is load-bearing.
  • HELD SCOPE: internal/eval/replay.go sandboxes stay in the system temp directory. They are the largest scratch this program creates, but a replay materializes its own nested NM_HOME and worktree while Store.Prune, the case records, and the object pools all live under <NM_HOME>/eval - so relocating the sandbox inside the app root nests it in the state it is replaying, which e2e TestEvalJourney refuses on purpose. Moving it needs a disk-backed root outside NM_HOME, which does not exist yet; do not "fix" it by weakening that assertion. Every remaining os.MkdirTemp("", ...) caller is auto-named and self-cleaning with defer; keep it that way.
  • Regressions: internal/paths/evidence_test.go, internal/config/evidence_storage_test.go, internal/daemon/evidence_reap_test.go, TestSteeringNamesTheConfiguredEvidenceRoot, TestTestEvidenceDir_DefaultResolutionStaysUnderTheAppRoot, e2e TestTestEvidenceLivesUnderAppRootNotSharedTemp / TestRunCleanupLeavesNoEmptyEvidenceDirectory.
Repository

kunchenguid/no-mistakes

Path

.agents/skills/test-evidence-storage/SKILL.md

Last updated
First committed

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.