CtrlK
BlogDocsLog inGet started
Tessl Logo

exploring-ai-failures

Find where an AI/LLM application is failing in production and surface the failure patterns, working from real traces. Use when someone wants to understand what's going wrong with an AI feature, find and categorize failure modes, triage errors, or investigate quality issues (wrong answers, ignored instructions, hallucinations, tool misuse) — "what's failing in my agent", "surface error patterns", "why are the responses bad", "find the common failure modes", "what should I fix next". Covers scoping to one use case, finding failing traces by whichever signal fits the context (code errors, metric outliers, trace-type slices, manual review, existing-eval spikes, clustering), and reading them into a ranked failure taxonomy.

75

Quality

92%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

85%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, actionable workflow skill with clear sequencing, explicit stopping/verification checkpoints, and good progressive disclosure. The main weakness is repetition of the central 'reading is the job' thesis across several sections.

Suggestions

Consolidate the 'reading is the job / loud minority' thesis: state it once in the intro and once in Step 3, then let Step 2's trap and the Tips reference it briefly rather than re-explaining it in full.

In Step 4's parenthetical '(A list assembled from error messages or metric counts you never read is the loud subset, not this — go back to Step 3.)', shorten to a one-line pointer to avoid re-deriving the same point made in Step 2's trap callout.

Trim restated guidance in the Tips section that duplicates Step 2/3 prose (e.g. 'Don't over-index on errors' restates the $ai_is_error point) to keep the closing section purely additive.

DimensionReasoningScore

Conciseness

Mostly lean with no concepts-Claude-knows padding, but the core thesis ('reading is the job' / the 'loud minority' trap) is restated across the intro, Step 2's trap callout, Step 3, Step 4's parenthetical, and the Tips, so it could be tightened.

2 / 3

Actionability

Gives concrete, copy-paste-ready tool invocations (e.g. generate-app-url with params), specific property names ($ai_is_error, $ai_trace_id), and concrete numbers ('roughly 20–30 traces') — highly actionable for an instruction-only skill.

3 / 3

Workflow Clarity

A clear 4-step sequence (Scope → Pick → Read → Rank/link) with an explicit stopping criterion ('keep reading until new traces stop turning up new modes') and a built-in user-verification feedback loop in Step 4.

3 / 3

Progressive Disclosure

Clear overview body with well-signaled, one-level-deep references to references/finding-traces.md (verified present) and clearly labeled sibling-skill references; detail is appropriately split out.

3 / 3

Total

11

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: concrete actions, abundant natural trigger terms, explicit what-and-when guidance, and a clearly distinct niche. It stays in third person and avoids fluff.

DimensionReasoningScore

Specificity

Lists multiple specific concrete actions — 'find and categorize failure modes, triage errors, investigate quality issues' and a concrete menu of trace-finding signals (code errors, metric outliers, trace-type slices, clustering) feeding a 'ranked failure taxonomy'.

3 / 3

Completeness

Explicitly answers both 'what' (find and surface production failure patterns from real traces) and 'when' via a 'Use when someone wants to…' clause with explicit triggers.

3 / 3

Trigger Term Quality

Embeds natural phrasings a user would actually say — 'what's failing in my agent', 'why are the responses bad', 'find the common failure modes', 'what should I fix next' — giving strong coverage of natural trigger terms.

3 / 3

Distinctiveness Conflict Risk

Carves a distinct failure-pattern-discovery niche ('ranked failure taxonomy', 'surface error patterns') and differentiates from sibling exploring-llm-* skills, making wrong-skill triggering unlikely.

3 / 3

Total

12

/

12

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository
PostHog/posthog
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.