phoenix-error-analysis

Find out what is going wrong in LLM or agent traffic by reading sampled Phoenix traces, spans, or sessions, writing free-form notes (open coding), then grouping the notes into a few narrow annotations, one per failure dimension with a small label set and counts, that pick eval targets and fix priorities (axial coding). Use for "what's going wrong with this agent", "I just instrumented my app, where do I start", "review these traces", "the chatbot keeps losing context", "what kinds of mistakes is the model making", "what categories of failures do we have", "what should I build evals for", "how do I prioritize fixes", "group these notes", "MECE breakdown" — or any framing that needs observations or categories grounded in real traces rather than invented top-down, even without naming the technique.

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

73%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, actionable workflow skill with clear sequenced phases, strong validation checkpoints, and sensible delegation of detail to sibling skills. Its main weakness is conciseness — repeated emphasis and duplicated warnings inflate the body.

Suggestions

Remove the duplicated 'do not sample by error status' guidance — state it once and cross-reference, to cut redundancy between phase 1 inspection and phase 2 entity selection.

Consolidate the repeated 'recorded notes are the deliverable, not a chat summary' emphasis into a single stated-once principle to reduce rhetorical padding.

Consider moving the full CLI/GraphQL operation table and the URL filter-expression tables into a reference file (or sibling-skill pointer) to slim the SKILL.md body further, keeping only the operation map inline.

Dimension	Reasoning	Score
Conciseness	Mostly procedural with commands, tables, and JSONL schemas, but it carries real redundancy — 'don't sample by error status' appears twice (phase 1 inspection and phase 2 entity selection) and 'notes are the deliverable, not a summary in chat' is emphasized repeatedly — so it could be tightened rather than being cleanly efficient.	3 / 5
Actionability	Provides copy-paste bash for the sidecar setup, concrete `px` CLI command templates with flags, exact JSONL row schemas, and exact URL filter expressions; the judgment steps (Group, Define) are necessarily instructional rather than code, leaving minor gaps.	4 / 5
Workflow Clarity	Numbered 6-step and 8-step phase sequences carry explicit validation checkpoints (sidecar uniqueness `exit 1`, absent-file 'stop and run phase 1 first', saturation stop rule, newest-ts-wins dedup), feedback loops for resume and unit-pivot, checklists, and explicit confirmation gating on the destructive discard — validation is present so the destructive cap does not apply.	5 / 5
Progressive Disclosure	Ships no own bundle files (confirmed) and deliberately delegates details one level deep to sibling skills (phoenix-cli, phoenix-graphql references) that are clearly signaled; section structure is good, with only minor inlined reference-style content (the operation table and URL filter patterns).	4 / 5
	Total	16 / 20 Passed

Description

96%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, concrete description that explicitly covers what the skill does and when to use it, with abundant natural trigger phrases. The only weakness is a broadening catch-all clause that slightly raises conflict risk with general analysis skills.

Dimension	Reasoning	Score
Specificity	Names multiple concrete actions — 'reading sampled Phoenix traces, spans, or sessions', 'writing free-form notes (open coding)', 'grouping the notes into a few narrow annotations... with counts, that pick eval targets and fix priorities (axial coding)' — covering the full workflow comprehensively, matching the score-5 anchor.	5 / 5
Completeness	Explicitly answers both 'what' (read traces, write notes, group into annotations with counts that pick evals/fixes) and 'when' ('Use for ...' followed by concrete trigger phrases), matching the score-5 anchor.	5 / 5
Trigger Term Quality	Packs in many natural user phrasings — 'what's going wrong with this agent', 'the chatbot keeps losing context', 'what kinds of mistakes is the model making', 'what should I build evals for', 'group these notes', 'MECE breakdown' — comprehensive natural-term coverage including synonyms.	5 / 5
Distinctiveness Conflict Risk	The Phoenix-trace-grounded open/axial coding niche is distinct, but the catch-all 'or any framing that needs observations or categories grounded in real traces... even without naming the technique' broadens it and creates minor overlap with generic analysis skills, so it is a notch below a clean 5.	4 / 5
	Total	19 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: Arize-ai/phoenix
Path: .agents/skills/phoenix-error-analysis/SKILL.md
Commit: e127482

Reviewed: 6 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.