Evidence Packs — Agent Surface

Every kane-cli run seals an evidence pack: one .evidence file (a zip) holding the test definition, results, per-step screenshots + annotated screenshots, per-step console/network logs, run logs, and failure records. It is the only place run artifacts live — the legacy runs/<n>/ session subdirectory is no longer created — and the primary artifact for showing a user what happened and for debugging.

Inside a pack

A .evidence file is a standard zip: unzip -l <pack> lists it, unzip -p <pack> <entry> prints one entry.

<execution_id>.evidence
├── run.yaml                          # run manifest: title, status, started/ended, totals
├── failure.yaml                      # run-level failure rollup
└── tests/<test-id>/                  # one per test (testrun packs have many)
    ├── test.md                       # the definition
    ├── result.yaml                   # verdict + steps[] (ordinal, status, kind, duration_ms, action_id)
    ├── logs/                         # meta.yaml, tui.log, and per run index n:
    │                                 #   <n>-run.log, <n>-actions.ndjson,
    │                                 #   <n>-console.ndjson, <n>-network.har
    ├── steps/<ordinal>-<step-id>/    # screenshot.png, annotated.png, step.json
    │                                 #   (+ failure.yaml on failed steps)
    └── …                             # the full execution record and planning notes (not needed to diagnose)

Reading and debugging from the pack (unzip recipes, failure records): references/debug.md.

Where packs land

The stderr hint — and how to act on it

After an agent-mode run whose evidence pack sealed (passed or failed), kane-cli prints one hint line to stderr (it is NOT an NDJSON event; never look for it on stdout):

evidence: view locally with `kane-cli evidence serve <packPath>`

On a person's first run, do not just offer: start the server and put the viewer link in the result card, so the tour's "evidence" becomes something they can click (references/first-run.md §3). From the second run on, go back to offering.

When you see it (or when the user asks to see run evidence): offer — "Want to view the run evidence in your browser?" If yes, run the serve command via Bash (run_in_background so it keeps serving) and give the user the viewer URL from its stdout:

serving 1 pack on http://127.0.0.1:<port>
<name>.evidence
  pack    http://127.0.0.1:<port>/<token>/<name>.evidence
  viewer  https://evidence.lambdatest.com/?pack=<encoded pack url>
press Ctrl-C to stop

Present only the viewer line as a clickable link. Tell the user the server is local-only (nothing uploads; their browser reads the pack from their machine) and stays up until stopped. Stop the background process when they're done.

Validate — integrity checks

kane-cli evidence validate <execution-id-or-path> --json

Exit 0 valid · 1 invalid · 2 not found. Use it when a pack won't open in the viewer (truncated / unsealed / killed run) or as a CI gate. --profile L0|L1 selects strictness (default L1).

Merge — many packs into one

kane-cli evidence merge <targets...> --run-id <id>

Combines packs (execution ids or paths, order-significant) into one, sealed by default at .testmuai/evidence/<run-id>.evidence. --run-id is required. Exit 0 merged · 1 merge-rules abort (e.g. packs from different projects or organisations, or a pack that isn't sealed and valid; a pack whose run id repeats is skipped, not an abort) · 2 usage. Full merge-rules detail is in the user guide — for anything beyond a simple merge, point the user there rather than improvising flags.

After 0.10.0-rc.3, --policy <file> / --policy-id <id> (repeatable) judge the sealed merged pack: exit 1 then also means a failing verdict (pack and verdict files written), 3 an errored verdict; --json adds verdict {…} to the report; --no-finalize with a policy is exit 2. See references/policy.md.

Verdict — judge a pack against a policy (after 0.10.0-rc.3)

kane-cli evidence lint-policy <policy.yaml> --json
kane-cli evidence verdict <execution-id|path|latest> --policy <policy.yaml> [--policy-id <id>] --json

Writes <pack>.verdicts/<policy-id>.verdict.json beside the pack (never inside it). Exit 0 pass · 1 fail · 3 errored · 2 usage or I/O. Policy format, metrics, the verdict event, run-time --policy and Test Manager policies: references/policy.md.

Debugging with a pack

The pack is the first place to look on a failure: the failed step's failure record (error + page state), its console/network slice (4xx/5xx or JS errors usually explain it), and the annotated screenshot (what the agent actually acted on). Full failure workflow: references/debug.md.

Evidence merge identity

For ordinary runs, default merge identity distinguishes the test, commit and dataset row (external_id.test_id, external_id.commit_id, external_id.dataset_row_id) and the environment (environment.os, environment.os_version, environment.browser, environment.browser_version, environment.device, environment.app). Re-runs with the same identity nest as attempts; a different environment produces a separate sibling, so one test on two devices, or on two builds with different app identities, is two entries (two local builds with the same file name can group together). On a real device, environment.device is the device the hub allocated (never the requested pattern) and device_slug is derived from it.

Explicit collision policies can change grouping; a custom --rules file replaces the default rules rather than extending them. Check the selected identity rules before interpreting two runs as retries of the same test.