CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

phoenix-error-analysis

Find out what is going wrong in LLM or agent traffic by reading sampled Phoenix traces, spans, or sessions, writing free-form notes (open coding), then grouping the notes into a few narrow annotations, one per failure dimension with a small label set and counts, that pick eval targets and fix priorities (axial coding). Use for "what's going wrong with this agent", "I just instrumented my app, where do I start", "review these traces", "the chatbot keeps losing context", "what kinds of mistakes is the model making", "what categories of failures do we have", "what should I build evals for", "how do I prioritize fixes", "group these notes", "MECE breakdown" — or any framing that needs observations or categories grounded in real traces rather than invented top-down, even without naming the technique.

Quality

91%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

81%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A strong operational skill body: two clearly sequenced phases with real validation checkpoints, feedback loops, and destructive-operation safeguards, plus concrete commands, schemas, and URL templates. The residual issues are trimming duplicated guidance and moving end-of-run details (wrap-up linking, discard) into a reference file, with one full GraphQL mutation example being the main executability gap.

Suggestions

State each repeated principle once and cross-reference it: the error-status warning (Phase 1 callout vs 'Choosing the level') and the 'invented top-down / notes are the deliverable' rationale (intro vs 'Recording notes' vs axial 'Group' step) each appear 2-3 times.

Move the wrap-up material — per-level filter expressions, URL-encode templates, span deep-linking, and the discard procedure — into a small reference file (e.g. wrapup.md), since it is needed only at run end and would slim the ~250-line body.

Add one complete, executable GraphQL mutation example (e.g. a full createTraceNotes query with variables) alongside the mutation-name table, since the GraphQL surface currently lists field and input names without a copy-paste-ready query.

DimensionReasoningScore

Conciseness

The body is dense with operational specifics (sidecar bash, JSONL schemas, operation-to-command table, URL templates) and teaches no background concepts, but a few points are repeated: the error-status warning appears both as the Phase 1 callout ("Don't sample by span status `ERROR`...") and again in Choosing the level ("Do **not** select entities by error status"), and the 'invented top-down' rationale is restated three-plus times (intro, "Recorded notes are the deliverable", "Work bottom-up from the note text"). These are minor instances of over-explanation that could be trimmed, fitting anchor 4 rather than the every-token-earns-its-place of anchor 5.

4 / 5

Actionability

Concrete, largely copy-paste-ready guidance throughout: a complete sidecar-setup bash block with an existence check, full command shapes ("px trace add-note <trace-id> --text \"...\" --identifier \"$CODING_ANNOTATION_IDENTIFIER\""), exact JSONL row schemas, example dimension YAML, and complete filter expressions for UI links. It falls short of anchor 5 only in minor gaps: the GraphQL column lists mutation and field names ("createTraceNotes ... plus an entity reference by OTel id") without one full executable query, and exact `px` list/sampling flags are delegated to the phoenix-cli skill.

4 / 5

Workflow Clarity

Both phases have numbered, explicitly sequenced processes (6 and 8 steps) with an explicit between-phases decision point, and validation is built in: the sidecar existence test that exits 1, saturation criteria, malformed-line handling ("fix or drop a malformed line without touching its neighbors"), newest-ts-wins resolution, "An absent file means open coding hasn't run ... stop and run phase 1 first", and destructive discard gated on explicit user confirmation with the delete-all flag. This matches anchor 5's explicit validation, error-recovery feedback loops, and checklists; the destructive/batch cap does not apply because validation and confirmation steps are present.

5 / 5

Progressive Disclosure

Structure is good and the skill appropriately delegates API detail to other skills' references ("The `phoenix-cli` skill documents the `px` flags, and the `phoenix-graphql` skill's references ... document the GraphQL fields"; "This skill ships no reference files of its own"), keeping the body an operations overview rather than an API dump. It stops short of anchor 5 because at ~250 lines with no reference files of its own, some late-stage operational detail (wrap-up URL/filter construction, discard procedure, resuming-a-run mechanics) is inlined where a one-level-deep reference file could keep the working body leaner.

4 / 5

Total

17

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: it states a concrete multi-step capability in third person, includes an explicit 'Use for' clause with ten realistic trigger phrases plus a catch-all, and occupies a distinct Phoenix-trace-analysis niche. The only conceivable risk is that a couple of the generic trigger phrases could fire in non-trace contexts, but the trace grounding throughout keeps that minimal.

DimensionReasoningScore

Specificity

"reading sampled Phoenix traces, spans, or sessions, writing free-form notes (open coding), then grouping the notes into a few narrow annotations, one per failure dimension with a small label set and counts, that pick eval targets and fix priorities (axial coding)" names multiple concrete actions with concrete artifacts (notes, annotations, label sets, counts) covering the full workflow end to end. It exceeds anchor 4 because there are no coverage gaps — reading, noting, grouping, annotating, counting, and prioritizing are all explicit.

5 / 5

Completeness

Both halves are explicit: the 'what' is the three-part open-coding/axial-coding workflow with its output, and a literal "Use for ..." clause lists concrete trigger phrases. This matches the anchor-5 example pattern (full capability statement + 'Use when' with concrete triggers), not the merely-explicit 'when' of anchor 4.

5 / 5

Trigger Term Quality

Ten quoted natural phrases users would actually say ("what's going wrong with this agent", "I just instrumented my app, where do I start", "review these traces", "the chatbot keeps losing context", "what should I build evals for", "how do I prioritize fixes", "group these notes", "MECE breakdown") plus a catch-all for unnamed framings. Coverage of natural phrasings and synonym framings is comprehensive, matching anchor 5 rather than the 'a few natural terms missing' of anchor 4.

5 / 5

Distinctiveness Conflict Risk

The description is anchored to a clear niche — Phoenix trace/span/session error analysis via grounded open/axial coding — and most triggers ("review these traces", "what's going wrong with this agent", "the chatbot keeps losing context") are unmistakably trace-review requests. A few listed phrases ("how do I prioritize fixes", "group these notes") are generic-sounding, but the catch-all bounds the skill to framings "grounded in real traces", keeping conflict risk minimal rather than the 'minor overlap with closely related skills' of anchor 4.

5 / 5

Total

20

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

Arize-ai/phoenix

Path

src/phoenix/server/mcp/skills/phoenix-error-analysis/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.