Content
85%Weight 40%Scale 1-5Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.
A strong, highly actionable triage workflow: concrete commands, exact file locations, a decision table, an explicit output format, and validation checkpoints with re-run feedback loops. The two real defects are the missing referenced file (reference.md) that dead-ends the only external pointer, and a duplicated command block in Rule 0.
Suggestions
Ship the referenced reference.md (worked triage examples for real regression vs test bug vs flaky judge) or remove the 'Additional resources' section — the link currently points to a file that does not exist in the bundle.
Delete the duplicated vitest command under "To reproduce judge graders locally" in Rule 0, or replace it with the genuinely different command it was presumably meant to hold.
Align step numbering/references: Step 4 says 'Re-run the single scenario (Step 0 command)' but the section is titled 'Rule 0', and the scenario-id placeholder is never tied to where scenario ids are listed (e.g., the scenarios/ directory or the -t match string).
| Dimension | Reasoning | Score |
|---|---|---|
Conciseness | The body is dense and lean — file-path tables, a symptom→verdict→fix-target decision table, and bare commands with no explanation of concepts Claude already knows. Not a 5 because of one clear redundancy: the "To reproduce judge graders locally" block in Rule 0 repeats the exact same vitest command already given two lines above, and "Step 0 command" in Step 4 refers to a section labeled "Rule 0" (a small naming mismatch). Not a 3: there is no unnecessary explanation or padding anywhere else. | 4 / 5 |
Actionability | Fully executable guidance throughout: a copy-paste-ready re-run command (`pnpm --filter @novu/agent-evals exec vitest run --config vitest.evals.config.ts -t <scenario-id>`), exact file paths for every grader layer (`src/suites/agent-onboarding/scenarios/<id>/graders.ts`, `catalog.ts`, `src/core/graders.ts`, `src/core/judge.ts`), the concrete pass threshold (`averages ≥ 0.8 (JUDGE_THRESHOLD)`, `UNKNOWN`→`skip` scores 1), a grader-kind decision table, and a fill-in-the-blanks output template. Not a 4: there are no gaps — the common triage case is covered end-to-end. | 5 / 5 |
Workflow Clarity | The sequence is explicit and gated: Rule 0 (re-run 3–5× to rule out flakiness before changing anything) → identify grader kind → read RunResult evidence → classify via top-down decision table → apply one bounded fix and verify. Validation checkpoints are explicit at every stage: "Confirm the fix holds across the 3–5 re-runs and that no other scenario regressed" and running the synthetic unit tests when a grader is edited. This matches the top anchor (feedback loops, explicit validation, ordered first-match classification). | 5 / 5 |
Progressive Disclosure | The body itself is well structured with clear sections, but the single external reference is broken: "For worked triage examples ... see [reference.md](reference.md)" points to a file that does not exist in the bundle (no references/ directory), so navigation dead-ends and the promised worked examples are unavailable. Not a 4: a clearly signaled but missing reference is a real navigation defect, not a minor organization gap; not a 2: the body is appropriately sized and organized rather than a monolith that should be split. | 3 / 5 |
Total | 17 / 20 Passed |