CtrlK
BlogDocsLog inGet started
Tessl Logo

exploring-llm-evaluations

Investigate AI observability evaluations — `hog` (deterministic code-based), `llm_judge` (LLM-prompt-based), and `sentiment` (user-message sentiment). Find existing evaluations, inspect their configuration, run them against specific generations, query individual results, and generate AI-powered summaries for boolean pass/fail runs. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, inspect sentiment classifications, or manage the evaluation lifecycle.

75

Quality

92%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

92%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, highly actionable skill body with executable examples, explicit verification steps, and clean navigation. The only weak spot is mild redundancy of the Hog-vs-LLM-judge decision guidance across several sections.

Suggestions

Consolidate the Hog-vs-LLM-judge guidance: keep the decision table and 'When to use' section, and have the investigation-pattern sections reference them rather than restating the rationale.

The lifecycle management table (Add/Tweak/Toggle/Disable/Remove) repeats actions already demonstrated with full payloads earlier; consider trimming it to a quick-reference or linking back.

Tighten repeated mentions of LLM-judge non-determinism and AI-data-processing approval, which appear in both the build/test workflow and the Tips section.

DimensionReasoningScore

Conciseness

Efficiently explains PostHog-specific concepts Claude would not know, but the Hog-vs-LLM-judge guidance is restated across the decision table, 'When to use' section, and investigation patterns, leaving minor redundancy that could be trimmed.

4 / 5

Actionability

Provides copy-paste-ready JSON tool invocations and exact-property SQL queries covering the common cases (find, summarize, drill in, verify, run, build/test both evaluator types, manage lifecycle).

5 / 5

Workflow Clarity

Numbered workflows include explicit validation checkpoints (e.g. Step 4 'Verify the pattern with raw SQL' with the N/A-guard rationale) and feedback loops (test-hog iterate-then-promote, LLM-judge create-disabled→run→refine→enable).

5 / 5

Progressive Disclosure

Clean section organization with one-level-deep, clearly signaled external references (tools.yaml, the exploring-llm-traces skill for URL conventions) and no nested-reference chains.

5 / 5

Total

19

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that clearly states capabilities and provides explicit trigger guidance across the evaluation lifecycle. The only mild gap is trigger-term breadth confined to one product's vocabulary.

DimensionReasoningScore

Specificity

Names the three evaluation sub-types and enumerates five concrete lifecycle actions (find, inspect configuration, run against generations, query results, generate summaries), matching the 'comprehensive coverage' anchor.

5 / 5

Completeness

Explicitly answers both what it does and when to use it, with a concrete 'Use when...' clause listing trigger phrases, matching the top anchor.

5 / 5

Trigger Term Quality

Includes many natural user phrases ('debug why an evaluation is failing', 'dry-run a Hog evaluator', 'prototype a new LLM-judge prompt') but stays within a single product's vocabulary without broader synonym coverage, so a few natural terms are missing.

4 / 5

Distinctiveness Conflict Risk

A narrow PostHog AI-observability niche with named sub-types (hog, llm_judge, sentiment) and distinct triggers, giving minimal conflict risk with other skills.

5 / 5

Total

19

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository
PostHog/posthog
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.