Diagnose Benchmark Failure

Why this skill exists

When a scenario fails, the natural reflex is to edit the primitive and re-run. That is guessing. A judged run leaves four artefacts behind — the judge's rendering, the raw transcript the agent's CLI wrote for itself, the snapshot of the primitive the agent read, and the scenario definition — and together they carry the actual chain of cause and effect: what the agent saw, what it did, what it emitted, what the judge measured. This skill enforces an evidence-first diagnosis so each iteration moves on facts, not hopes.

Harnesses differ. Names and paths in this file are examples; the four artefact kinds are the constant. Match by kind, never by file name.

Rules

Inputs

• A scenario ID, or the path of the run directory. Inferred from the user prompt or from the most recent failure in the harness's output.

Step-by-step

<step_by_step>

1. Locate the run and its four artefacts

   • If this project ships an addendum that states its own layout — a project-level skill, or a section of the project instructions naming the run directory and the artefact file names — read it first and use its paths. It saves the search and it is authoritative where it applies.
   • Otherwise search. Find the directory holding the run for this scenario (usually the most recently modified one whose path carries the scenario ID), then identify inside it:
     • (a) the judge's rendering — a text or Markdown file holding the prompt the agent received, a log of the session, and the file diffs.
     • (b) the raw transcript — the JSON-lines file the sandboxed CLI wrote for itself: find <run-dir> -name '*.jsonl'.
     • (c) the primitive snapshot — the copy of the failing primitive inside the run's workspace, under the IDE's config directory (.claude/skills/, .codex/skills/, .cursor/skills/, .opencode/skills/).
     • (d) the scenario definition — the source file declaring the prompt, the simulated user and the checklist.
   • A run may hold several attempts (run-1, run-2, attempt-1). Diagnose each failed attempt, not only the first.
   • A kind you cannot find after searching → fail closed (rule 4).

2. Read the judge's rendering end to end

   • Find its three parts, whatever they are labelled: the prompt the agent received, the log of the session, the diffs of the files it changed.
   • From the session log, extract the last assistant turn the user would have seen. This is the agent's actual emitted output.
   • Copy the prompt verbatim.
   • Note any tool calls the rendering shows. Treat this list as provisional: a rendering can omit calls, and step 3 replaces it with the count from the transcript.

3. Read the raw agent transcript

   • Print the tool-call histogram BEFORE forming any hypothesis. Two common CLI formats:
     • codex rollout: jq -r 'select(.type=="response_item") | .payload | select(.type=="function_call" or .type=="custom_tool_call") | .name' <file> | sort | uniq -c | sort -rn (arguments are in .payload.arguments / .payload.input; the agent's own text is in response_item lines of type message with role assistant, its reasoning in lines of type reasoning).
     • claude transcript: jq -r 'select(.message.content|type=="array") | .message.content[] | select(.type=="tool_use") | .name' <file> | sort | uniq -c | sort -rn (the agent's own text and thinking are assistant lines whose content blocks are of type text / thinking). Another CLI needs another selector; read one line of the file and write the equivalent query rather than skipping the histogram.
   • Copy the histogram into the report verbatim. Then locate the decisive call — the invocation, or the absence of one, that the failing checklist item is about — and quote the agent's own text or reasoning around it: that is where the agent explains its choice to itself, and that is what the judge's rendering never shows.
   • Cross-check every capability claim in the session log against the histogram (rule 5). Record each as "claimed X; transcript shows Y".

4. Read the scenario definition

   • Extract: the prompt given to the agent, the simulated user's persona, and the checklist items with their ids, descriptions and critical flags.
   • Also extract whatever the harness lets a scenario stage before the run: its setup step, its template variables, whether the run was interactive.

5. Read the primitive — BOTH copies, side by side

   There are two different copies of the same primitive, in two different locations. You MUST read both and compare them. Confusing them leads to the wrong classification.

   • (a) The failing agent's view: the snapshot inside the run's workspace. This is the static copy the failing agent read. Read it first.
   • (b) The current source: the file in the project tree that the next edit would change. It may differ from (a).

   Do NOT read the project's own installed copy — the one the IDE loads for your session — and call that the snapshot. It is the current source under another path, and it tells you nothing about what the failing agent saw.

   After reading both, diff (a) vs (b):

   • Identical → the failure cannot be a stale-copy issue; eliminate STALE-SKILL-IN-SANDBOX from candidates.
   • Significantly different → potential STALE-SKILL-IN-SANDBOX; verify by correlating the diff with the failure mode.

   The classification depends on what (a) said vs. what the agent actually emitted — not on what (b) currently says.

6. Re-derive the verdict

   • Which checklist items failed and why is usually in the harness's output, not in the judge's rendering. If you don't have it, re-run the scenario with the harness's cache disabled and capture its output. Otherwise, check each checklist description against the agent's last turn from step 2 and judge yourself before continuing — this catches LLM-judge calibration drift.

7. Match symptoms to the failure-mode taxonomy (next section). Pick the most likely mode based on the quoted evidence. If two modes fit equally, list both; do not collapse them.

8. Decide whether an interview is the next evidence step

   • The transcript shows what the agent DID; only the agent can say which words it justified the act with, and those are the words a fix would edit. So when the fix-direction you are about to propose is a change to the primitive's wording (a sentence to add, tighten, or remove), the report MUST name resuming the failed session and asking it why as the next evidence step, before any edit — and give the command for the run's CLI. The session's home directory and working directory are the ones the run used, both preserved in the run directory:
     • codex: cd <run workspace> && CODEX_HOME=<run home>/.codex codex exec resume <uuid> "<question>" (<uuid> is the tail of the rollout filename; open the question with "Do not invoke any skill; answer from memory").
     • claude: cd <run workspace> && HOME=<run home> claude -p --resume <uuid> "<question>" (<uuid> is the transcript filename without its extension). Resolve a symlinked run directory with readlink before using it as a path. The workspace outlives the run, so the session can be resumed in place.
   • Draft the question neutrally: describe the situation without accusing; ask what made the chosen path better than the alternative, which phrase in the rule left room for it, and what the rule would have had to say. Say it is to be asked of EVERY failed run of the scenario — agreement across runs is what separates a defect in the text from one agent's rationalisation.
   • When the fix-direction is not a wording change (a harness defect, a stale snapshot, a primitive that was never installed), say so and skip the interview.

9. Write the diagnostic report (template below). Every claim cites a quoted line from step 2/3/4/5.

</step_by_step>

Failure-mode taxonomy

A symptom-to-cause map. Use the symptom column to match what you observed; the cause column gives the most likely root cause; the fix-direction column points the next iteration at a real lever, not a guess. Do not invent new modes unless the evidence rules out every one listed.

• MD-PRIOR-BULLETS (markdown-prior-wins-over-instruction)

  • Symptom: the primitive says options must be numbered (1., 2., …); the agent emits bulleted dashes (- **X** —) for option lists with rich descriptions.
  • Cause: the model's training prior on bulleted-options-with-descriptions overrides plain-prose MUST instructions in the primitive's context.
  • Fix-direction: scope the requirement down to what is enforceable through the primitive's text alone, OR add a runtime mechanism (a helper script the agent invokes, a post-process hook). Adding more imperative prose has been empirically rejected for this mode.

• HEADING-INSTEAD-OF-ITEM

  • Symptom: the primitive says the question must be a numbered list item; the agent emits ### Variant A, ### Variant B, or **1. Title** (bold heading).
  • Cause: the instruction was read but not internalised at format time; the agent reverts to its default for "labelled chunks of related content".
  • Fix-direction: add an explicit anti-pattern example showing that ### Variant A is wrong; demonstrate the correct shape with the EXACT surface form expected.

• STALE-SKILL-IN-SANDBOX

  • Symptom: the agent's behaviour matches an older version of the primitive; the snapshot (c) diffs against the current source.
  • Cause: the harness reused a cached verdict for a scenario whose primitive was edited after the cached run, or the cache was not disabled on a quick re-run.
  • Fix-direction: re-run with the harness's cache disabled. If the issue persists, compare what the cache key covers against what actually changed.

• SKILL-NOT-MOUNTED

  • Symptom: the agent exits successfully but took zero steps, or the judge reports "Unknown skill" / the agent never invokes the primitive.
  • Cause: the harness did not copy the primitive into the run's workspace config directory — most often the scenario names a primitive that does not exist, or the set the harness copied does not include it.
  • Fix-direction: fix the harness or the scenario's primitive name, NOT the primitive's text.

• COMPOSITE-DELEGATION-BYPASS

  • Symptom: a composite primitive (one that inlines the work of several others) was invoked, but the session log shows an early delegation call re-entering one of the inlined sources, bypassing the composite's gate.
  • Cause: the composite's frontmatter description names the source primitives, or its body lacks a "no-delegation" rule.
  • Fix-direction: rename the description so it carries no source names, add the no-delegation rule, harden the verdict gate. Follow the project's own composite-authoring guidance where it has one.

• PERSONA-MISMATCH

  • Symptom: the agent asks one question; the simulated user answers something the agent did not ask. The session log shows a reply that does not fit — whatever marker the harness uses for simulated-user input.
  • Cause: the persona was scripted for an older version of the primitive, or for a different question structure.
  • Fix-direction: align the persona with the primitive's current flow — but ONLY after confirming the primitive itself is correct. Persona changes that smuggle in the right answer ("test-fitting") are forbidden.

• TEST-FITTING-PERSONA

  • Symptom: the persona contains the literal target output (for example it dictates the exact format). The scenario passes only because the persona scripts the answer.
  • Cause: the scenario author worked backwards from a green run.
  • Fix-direction: rewrite the persona to be neutral. The scenario's diagnostic value is now suspect — review the checklist items too.

• CAPABILITY-CLAIMED-UNAVAILABLE

  • Symptom: the agent's own prose in the session log says a tool or capability was missing ("no subagent tool", "parallel execution unavailable") and the work was done another way; the transcript histogram shows that tool invoked (or installed in the workspace) in the same session.
  • Cause: the agent abandoned the approach — a worker's partial output looked slower to reconcile than redoing it, or the first call returned something awkward — and explained the retreat as an environment limit, licensed by a clause in the primitive such as "if X is unavailable, proceed with Y".
  • Fix-direction: interview the failed runs first (step 8); then close the clause that licensed the fallback so "unavailable" means "the tool is not in the tool list", not "the first attempt was inconvenient". Do NOT fix the harness: the capability was there.

• CROSS-PACK-REFERENCE-MISSING

  • Symptom: the primitive's text references another primitive by name, but that one is not in the workspace. The agent reads the reference and cannot act on it.
  • Cause: the referenced primitive ships in a group the scenario does not install; the harness's output usually names what it copied.
  • Fix-direction: drop the cross-reference, OR make the scenario install the group that carries it (rare; usually drop the reference).

Output template

Produce exactly this structure. Every bullet ends with a (<file>:<line-range>) citation.

# Diagnostic Report: <scenario-id>

## Run inspected
- Run dir: <path>
- Verdict line: "<paste>"
- Failed checklist items (id + critical?): <list>

## Evidence collected (paths)
- judge's rendering — <path>, <bytes>, <line count>
- raw transcript — <path>; tool-call histogram:
  <paste the histogram verbatim>
- primitive snapshot — <path>, <bytes>
- scenario definition — <path>

## Judge's rendering vs transcript
- <each capability or "I did X" claim from the session log: "claimed …; transcript shows …" with the decisive line quoted> (<transcript path>:<line>)

## Agent's last assistant turn (verbatim, ≤30 lines)

What the primitive actually said about this point (verbatim, ≤15 lines)

<paste>

(:-)

Diff: primitive snapshot vs current source

• <"identical" or summary of significant diffs with line refs>

Failure-mode classification

• Primary:
• Why this code: <one-sentence reason citing two of the four artefacts, the transcript among them when a capability claim is involved>
• Alternatives ruled out: <code(s) considered + the evidence that rules them out>

Proposed next iteration

• <one-sentence fix-direction action, drawn from the taxonomy fix-direction column>
• Files to edit:
• Interview: <"required — wording change" + the resume command with this run's uuid + the question, to be asked of every failed run | "not needed — ">
• Anti-actions (do NOT try): <bullets, each with the prior failure that disqualified it if applicable>

Confidence

• <High|Medium|Low> —