Name: exploring-llm-traces
Rating: 76 (1 reviews)
Author: PostHog

exploring-llm-traces

ABSOLUTE MUST to debug and inspect LLM/AI agent traces using PostHog's MCP tools. Use when the user pastes a trace or session URL (e.g. /ai-observability/traces/<id> or /ai-observability/sessions/<id>), asks to debug a trace, figure out what went wrong, check if an agent used a tool correctly, verify context/files were surfaced, inspect subagent behavior, investigate LLM decisions, or analyze token usage and costs. Also use when raw SQL/HogQL against `events.properties.$ai_input` / `$ai_output_choices` returns empty — message content lives only on the dedicated `posthog.ai_events` table.

Quality

93%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

92%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill body with executable examples, a clear sequenced debugging workflow, and clean progressive disclosure to verified reference files and scripts. The main improvement opportunity is deduplicating guidance repeated across the workflow, link-construction, and tips sections.

Suggestions

Consolidate the repeated '_posthogUrl always include' directive and date-range guidance so each appears once in its canonical section rather than re-stated in Step 2, 'Constructing UI links', and 'Tips'.

Merge the schema-discovery-first guidance, which is explained nearly identically in both the 'Finding traces' intro and the 'Discovering the schema first' subsection.

Dimension	Reasoning	Score
Conciseness	Content is dense, domain-specific, and avoids explaining general concepts Claude already knows, but several points are repeated across sections (the '_posthogUrl always include' directive, date-range guidance, and schema-discovery-first rule each appear multiple times) and could be tightened.	4 / 5
Actionability	Provides fully executable, copy-paste-ready JSON tool-call payloads, bash script invocations with env vars, and concrete filter examples covering trace URLs, session URLs, model/person/external-ID filters, with only placeholder IDs to substitute.	5 / 5
Workflow Clarity	The URL-debug workflow is a clearly sequenced three-step process with explicit decision logic (trace vs session classification, date_from/date_to/timestamp handling) and a validation guardrail ('CRITICAL: Never assume event names... call read-data-schema first') before constructing filters.	5 / 5
Progressive Disclosure	SKILL.md acts as a well-signaled overview with one-level-deep markdown links to real bundle files (./references/events-and-properties.md, example-llm-trace.md.j2, example-llm-traces-list.md) and a scripts table linking to six existing ./scripts/*.py files, keeping bulk detail off the main page.	5 / 5
	Total	19 / 20 Passed

Description

95%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, highly trigger-rich description that clearly defines a narrow PostHog LLM-trace debugging niche with comprehensive natural language triggers. The only blemish is the 'ABSOLUTE MUST to' opening, which reads as buzzword over-claim rather than clean third-person declarative voice.

Suggestions

Replace the 'ABSOLUTE MUST to debug and inspect' opening with a clean third-person declarative such as 'Debugs and inspects LLM/AI agent traces using PostHog's MCP tools' to remove the over-claim buzzword.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions (debug, inspect, 'check if an agent used a tool correctly', 'verify context/files were surfaced', 'inspect subagent behavior', 'analyze token usage and costs'), but the opening 'ABSOLUTE MUST to' is an over-claim buzzword that slightly muddies an otherwise concrete action list.	4 / 5
Completeness	Explicitly answers both what ('debug and inspect LLM/AI agent traces using PostHog's MCP tools') and when (an extensive 'Use when...' clause with multiple concrete triggers and a SQL/HogQL fallback condition).	5 / 5
Trigger Term Quality	Comprehensive natural trigger phrases a user would actually say ('pastes a trace or session URL', 'asks to debug a trace', 'figure out what went wrong', 'analyze token usage and costs') plus concrete URL path examples and synonyms.	5 / 5
Distinctiveness Conflict Risk	Clear niche (PostHog AI observability trace/session debugging via MCP) with distinct triggers like specific UI URL paths and the HogQL-returns-empty condition, giving minimal conflict risk with other skills.	5 / 5
	Total	19 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: PostHog/posthog
Path: products/ai_observability/skills/exploring-llm-traces/SKILL.md
Commit: 6fca5f8

Reviewed: 3 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.