triage-agent-eval-failures

Triage failing @novu/agent-evals scenarios to decide whether a failure is real or flaky, and whether to fix the playbook/prompt or the test (grader, tape, scenario, or judge). Use when an agent-evals scenario fails, when the user asks why an eval is red, or when deciding whether to fix the test or the prompt.

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

77%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable triage workflow with strong sequencing and validation checkpoints. Its main weaknesses are a duplicated command block and a referenced reference.md file that is not actually shipped.

Suggestions

Remove the duplicated command block: the 'reproduce judge graders locally' section repeats the Rule 0 re-run command verbatim — collapse to a single invocation or a cross-reference.

Ship reference.md in a references/ directory (with the worked real-regression / test-bug / flaky-judge examples) or drop the dangling link so navigation is not broken.

Consider moving the grader-location and RunResult-field tables into reference.md to keep SKILL.md as a tighter overview, improving progressive disclosure.

Dimension	Reasoning	Score
Conciseness	The body is lean and assumes competence (no concept explanations, dense path/field references), but the Rule 0 re-run command and the 'reproduce judge graders locally' command are verbatim duplicates ~8 lines apart, so it could be tightened.	2 / 3
Actionability	Provides exact executable pnpm/vitest commands, a precise file-location table, a RunResult field-to-check mapping, a symptom-to-verdict decision table, and a copy-paste output template.	3 / 3
Workflow Clarity	Sequences Rule 0 through Step 4 with explicit validation checkpoints (re-run 3-5x, confirm across re-runs, run synthetic unit tests after grader edits, check for regressions) and clear feedback loops.	3 / 3
Progressive Disclosure	Sections are well-organized and the reference to reference.md is one-level-deep and clearly signaled, but reference.md is not present in any bundle directory, leaving a dead navigation target.	2 / 3
	Total	10 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that states concrete capabilities and provides explicit, natural trigger conditions. It cleanly answers what the skill does and when to use it with minimal fluff.

Dimension	Reasoning	Score
Specificity	Lists multiple specific concrete actions: triage a failing scenario, classify as real vs flaky, and choose a fix target across grader/tape/scenario/judge — not just a domain label with partial actions.	3 / 3
Completeness	Explicitly answers both what (triage + classify + pick fix target) and when via a dedicated 'Use when...' clause, so the when is not merely implied.	3 / 3
Trigger Term Quality	The 'Use when...' clause surfaces natural phrasings an engineer would actually say ('scenario fails', 'why an eval is red', 'fix the test or the prompt') with good coverage.	3 / 3
Distinctiveness Conflict Risk	Scoped to @novu/agent-evals triage with distinct triggers, making it unlikely to fire for unrelated skills.	3 / 3
	Total	12 / 12 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
relative_links	Relative link issues: 1 missing	Warning

	Total	15 / 16 Passed

Repository: novuhq/novu
Path: .cursor/skills/triage-agent-eval-failures/SKILL.md
Commit: d921755

Reviewed: about 13 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.