CtrlK
BlogDocsLog inGet started
Tessl Logo

exploring-llm-evaluations

Investigate AI observability evaluations — `hog` (deterministic code-based), `llm_judge` (LLM-prompt-based), and `sentiment` (user-message sentiment). Find existing evaluations, inspect their configuration, run them against specific generations, query individual results, and generate AI-powered summaries for boolean pass/fail runs. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, inspect sentiment classifications, or manage the evaluation lifecycle.

72

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

77%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with clear, validated workflows, but it is a long monolithic single file with repeated Hog-vs-LLM-judge rationale and inline detail that could be offloaded to reference files. Conciseness and progressive disclosure are the weaker dimensions.

Suggestions

Collapse the duplicated Hog-vs-LLM-judge guidance: keep the comparison table once and have the intro and 'Investigation patterns' sections refer to it rather than restating the rationale.

Move the full event-schema table and the reference SQL queries into a references/ file (e.g. REFERENCE.md), keeping SKILL.md an overview that links to it — this would lift progressive_disclosure toward the score-3 anchor.

Trim the prose around tool-call examples that restates what the JSON already shows, to tighten conciseness toward the score-3 anchor.

DimensionReasoningScore

Conciseness

Mostly efficient (concrete tool calls, schema tables, SQL) but padded with redundancy — the Hog-vs-LLM-judge rationale is restated in the intro, the dedicated comparison table, and again under 'Investigation patterns', so it could be tightened; not lean enough for the score-3 anchor.

2 / 3

Actionability

Every tool is shown with a complete, copy-paste-ready JSON invocation, plus full SQL queries and concrete create/update config blocks — fully executable, not pseudocode, matching the score-3 anchor.

3 / 3

Workflow Clarity

Workflows are numbered with explicit validation checkpoints — Step 4 'Verify the pattern with raw SQL' noting 'the summary is LLM-generated and should be verified', plus iterate→promote feedback loops in the build/test workflow — matching the clear-sequence-with-validation anchor.

3 / 3

Progressive Disclosure

No bundle files exist and the 400+ line SKILL.md keeps everything inline — full event-schema table, comparison table, and multiple full SQL queries are content that could be split into references; well-organized into sections but 'content that should be separate is inline', matching the score-2 rather than the score-3 anchor.

2 / 3

Total

10

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is strong: it names concrete capabilities, supplies an explicit 'Use when...' trigger clause rich in natural user phrasings, and occupies a clear, low-conflict niche. No reductions apply under the rubric anchors.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — 'Find existing evaluations, inspect their configuration, run them against specific generations, query individual results, and generate AI-powered summaries' — matching the 'multiple specific concrete actions' anchor; not vague.

3 / 3

Completeness

Clearly answers both what (investigate the three evaluation types and the five lifecycle actions) and when via an explicit 'Use when...' trigger clause, matching the 'both what AND when with explicit triggers' anchor; not just implied.

3 / 3

Trigger Term Quality

The 'Use when...' clause surfaces natural user phrasings — 'debug why an evaluation is failing', 'surface common failure modes', 'compare results across filters', 'dry-run a Hog evaluator', 'prototype a new LLM-judge prompt', 'inspect sentiment classifications' — covering the ways a user would actually phrase the need.

3 / 3

Distinctiveness Conflict Risk

Niche is sharply defined — PostHog AI observability evaluations on the `posthog:llma-evaluation-*` tool family — with distinctive triggers (Hog dry-run, LLM-judge prompt prototyping) unlikely to fire for unrelated skills; the only cross-reference is an appropriately distinguished sibling skill.

3 / 3

Total

12

/

12

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository
PostHog/posthog
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.