CtrlK
BlogDocsLog inGet started
Tessl Logo

exploring-ai-failures

Find where an AI/LLM application is failing in production and surface the failure patterns, working from real traces. Use when someone wants to understand what's going wrong with an AI feature, find and categorize failure modes, triage errors, or investigate quality issues (wrong answers, ignored instructions, hallucinations, tool misuse) — "what's failing in my agent", "surface error patterns", "why are the responses bad", "find the common failure modes", "what should I fix next". Covers scoping to one use case, finding failing traces by whichever signal fits the context (code errors, metric outliers, trace-type slices, manual review, existing-eval spikes, clustering), and reading them into a ranked failure taxonomy.

73

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

85%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, mostly lean procedural skill with a clear sequenced workflow, explicit checkpoints, and clean progressive disclosure into a single real reference file. Its main weakness is mild repetition of the 'reading is the job' message across multiple sections.

Suggestions

Consolidate the repeated 'reading is the job / don't stop at aggregates' message — it appears in the intro, the Step 2 trap callout, Step 3, Step 4, and the Tips; stating it once prominently and referencing it would tighten conciseness.

Inline one or two of the most-used SQL query shapes (e.g. the code-errors sweep) directly in the body so the common path is executable without opening finding-traces.md.

The Tools table could note which tool each Step uses (e.g. map Step 2 strategies to query tools), making the workflow-to-tool linkage explicit without extra prose.

DimensionReasoningScore

Conciseness

The body is largely lean and assumes competence (e.g. the tight Tools table and Step 3's direct 'this is the job' framing), but the repeated insistence on 'reading is the job' and the trap/warning paragraphs re-state the same point several times and could be tightened.

4 / 5

Actionability

Concrete, executable guidance is present — specific tool calls, exact generate-app-url invocations with params, and real SQL in the referenced file — but the SKILL.md body itself leans on instruction with many specifics deferred to finding-traces.md rather than inlined, leaving minor gaps in copy-paste-ready detail in the main file.

4 / 5

Workflow Clarity

The four-step sequence (Scope → Pick traces → Read a batch → Rank/link/hand back) is clearly numbered, each step has an explicit checkpoint ('stop when it goes quiet', 'until new traces stop turning up new modes'), and it includes a feedback/recovery loop ('go back to Step 3' if you ranked from unread counts), plus explicit validation against misreading traces.

5 / 5

Progressive Disclosure

SKILL.md is a clear overview with one-level-deep, well-signaled references — the query detail lives in references/finding-traces.md (a real file), the event schema is delegated to exploring-llm-traces, and the body stays navigable with clean section headers.

5 / 5

Total

18

/

20

Passed

Description

96%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, well-crafted description that crisply states what the skill does, when to use it, and the natural trigger phrases a user would say. It is comprehensive and concrete with only minor overlap risk against sibling trace-analysis skills.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — 'find and categorize failure modes, triage errors, or investigate quality issues' plus scoped sub-activities like 'scoping to one use case, finding failing traces... and reading them into a ranked failure taxonomy' — giving comprehensive coverage of what the skill does.

5 / 5

Completeness

Explicitly answers both 'what' (find/surface/categorize/rank failure patterns from real traces) and 'when' with a full 'Use when...' clause listing concrete trigger scenarios, satisfying the highest anchor.

5 / 5

Trigger Term Quality

Packs in natural user phrases as direct triggers — 'what's failing in my agent', 'why are the responses bad', 'find the common failure modes', 'what should I fix next' — alongside explicit symptom terms (hallucinations, tool misuse, ignored instructions), covering synonyms and phrasings users actually say.

5 / 5

Distinctiveness Conflict Risk

The AI-failure-discovery niche is clearly framed and distinguishes itself from sibling skills via scoping/trace-reading language, but it sits alongside closely related skills (exploring-llm-traces, exploring-llm-clusters) creating minor overlap risk rather than a fully isolated niche.

4 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository
PostHog/posthog
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.