CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

phoenix-error-analysis

Find out what is going wrong in LLM or agent traffic by reading sampled Phoenix traces, spans, or sessions, writing free-form notes (open coding), then grouping the notes into a few narrow annotations, one per failure dimension with a small label set and counts, that pick eval targets and fix priorities (axial coding). Use for "what's going wrong with this agent", "I just instrumented my app, where do I start", "review these traces", "the chatbot keeps losing context", "what kinds of mistakes is the model making", "what categories of failures do we have", "what should I build evals for", "how do I prioritize fixes", "group these notes", "MECE breakdown" — or any framing that needs observations or categories grounded in real traces rather than invented top-down, even without naming the technique.

75

Quality

94%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

88%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A strong, unusually rigorous workflow document: both phases are precisely sequenced with validation checkpoints, destructive operations are confirmation-gated, and nearly every instruction is executable as written (exact commands, schemas, filter expressions, URLs). Its weaknesses are duplication of the anti-top-down and ERROR-status warnings, and a length that makes SKILL.md the full manual rather than an overview split across reference files.

Suggestions

Deduplicate the guidance stated twice: keep the 'do not sample/filter by span status ERROR' blockquote in Phase 1 and reduce Phase 2's 'Choosing the level' to a one-line cross-reference; similarly consolidate the anti-top-down warning, which currently appears in four to five near-identical forms (intro, Phase 1 saturation, Group step, Shaping the dimensions, What makes a good dimension).

Move the Wrapping-up URL mechanics (the per-tab search-param table, filter-expression templates, and URL-encoding rules, ~25 lines) into a reference file so SKILL.md reads as an overview pointing to detail, trimming token cost for invocations that end before wrap-up.

DimensionReasoningScore

Conciseness

The body is dense and operational throughout — it never explains concepts Claude already knows (no 'what is a trace/OTel' padding) and consistently gives mechanics over background. However, the anti-top-down warning recurs in near-identical forms four to five times ('dimensions and labels that are not grounded in notes are invented top-down, which is the failure this workflow exists to avoid', 'the top-down shortcut this workflow exists to replace', 'Work bottom-up from the note text; do not start from a list of dimensions', 'Grounded in data — emerged from actual note text, not assumed upfront'), and the 'do not select by span status ERROR' guidance is stated twice in full (the Phase 1 blockquote and Phase 2's Choosing the level). This is 'efficient with minor instances that could be trimmed' (4), not 'every token earns its place' (5).

4 / 5

Actionability

Fully executable guidance throughout: a copy-paste bash block for identifier/sidecar setup with an existence check, exact JSONL row schemas for both sidecars, concrete CLI commands with flags ('px trace add-note <trace-id> --text "..." --identifier ...'), named GraphQL mutations with their input shapes ('{ note, annotatorKind: LLM, source: API, identifier }'), exact upsert keys, exact filter expressions, and URL templates with encoding rules. The good/bad note and dimension examples cover the common cases; deferring full GraphQL/CLI details to named sibling skills is explicitly declared, so no gap is left implicit.

5 / 5

Workflow Clarity

Both phases have numbered processes with explicit validation checkpoints: sidecar existence/uniqueness check before recording, malformed-line handling ('fix or drop a malformed line without touching its neighbors'), newest-ts-wins reconciliation matching server upserts, saturation stop criteria, an explicit decision gate between phases (proceed vs. offer-and-wait), and resuming rules across invocations. The destructive discard operation is correctly gated on explicit user confirmation with per-kind deletes — the batch/destructive validation cap does not apply, and error-recovery loops (append a disambiguator and re-check, add a new label to its config before writing with it) are present throughout.

5 / 5

Progressive Disclosure

No bundle files exist, and the skill declares 'This skill ships no reference files of its own' — all external pointers are one level deep and clearly signaled (the named phoenix-cli and phoenix-graphql skills with their specific reference files). In-file structure is strong (clear section progression, tables, checklists), but at ~240 lines the body is a complete manual rather than an overview, and operational reference material (the write-surfaces table, the wrapping-up URL/filter-expression mechanics) would live in a reference file in a bundle that had one. This sits between the 4 anchor ('good structure; most content appropriately placed; minor organization gaps') and the 5 anchor ('overview pointing to detailed materials'), landing on 4.

4 / 5

Total

18

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: it names the domain, both phases, and the deliverable concretely, then gives a long, natural 'Use for' clause covering direct requests, adjacent framings, and a catch-all for users who don't name the technique. Its only cost is density — it is a long single sentence — but every clause is informative rather than padded, so no dimension is penalized.

DimensionReasoningScore

Specificity

Lists multiple concrete actions with comprehensive coverage of the workflow: 'reading sampled Phoenix traces, spans, or sessions, writing free-form notes (open coding)', 'grouping the notes into a few narrow annotations, one per failure dimension with a small label set and counts', and outputs that 'pick eval targets and fix priorities (axial coding)'. It covers input data, both phases, and the deliverable, matching the 5 anchor rather than the 4 anchor ('several specific actions; minor gaps') — no coverage gap is evident.

5 / 5

Completeness

Explicitly answers both 'what' (the full read-traces → open-code-notes → axial-annotate → count workflow in the first sentence) and 'when' (an explicit 'Use for' clause with ten concrete trigger phrases plus a catch-all framing). This matches the 5 anchor's example structure; the 'Use when...' guideline cap does not apply since the trigger clause is present and specific.

5 / 5

Trigger Term Quality

Comprehensive coverage of natural user phrasings including synonyms and framings: "what's going wrong with this agent", "review these traces", "the chatbot keeps losing context", "what kinds of mistakes is the model making", "what categories of failures do we have", "what should I build evals for", "how do I prioritize fixes", "group these notes", "MECE breakdown". These read as things a user would actually say, matching the 5 anchor; nothing common is missing.

5 / 5

Distinctiveness Conflict Risk

Clear niche: error analysis of instrumented LLM/agent traffic in Phoenix ('Phoenix traces, spans, or sessions', open/axial coding), with triggers anchored to instrumentation ('I just instrumented my app, where do I start', 'review these traces'). A couple of phrases ('what should I build evals for') could co-occur with an eval-design skill, but the grounding in real traces is stated explicitly, so overlap risk is minimal — 5 rather than 4 ('minor overlap risk') is justified by how strongly the triggers presuppose trace data.

5 / 5

Total

20

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

Arize-ai/phoenix

Path

src/phoenix/server/mcp/skills/phoenix-error-analysis/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.