exploring-llm-evaluations

Investigate AI observability evaluations — `hog` (deterministic code-based), `llm_judge` (LLM-prompt-based), and `sentiment` (user-message sentiment). Find existing evaluations, inspect their configuration, run them against specific generations, query individual results, and set up scheduled reports on an evaluation. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, inspect sentiment classifications, or manage the evaluation lifecycle.

Quality

89%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A strong, highly actionable body with concrete tool calls and SQL for every workflow and genuine feedback loops on the iterative build paths. Its main weakness is structure: it is a long single-file document that has not been split into reference bundles, which is the natural next step for progressive disclosure.

Suggestions

Extract the reusable HogQL recipes (pass/fail/N/A breakdown, failing-run sampling, daily-fail regression, N/A audit) into a references/recipes.md and link to them from each workflow to reduce inline repetition.

Move the full event-schema property table into a references/event-schema.md, keeping only the most-used properties inline in SKILL.md.

Add an explicit validation/confirmation checkpoint to the destructive paths (e.g. confirm evaluation name/UUID before llma-evaluation-delete, and verify enabled state before flipping llma-evaluation-update enabled:true in production).

Dimension	Reasoning	Score
Conciseness	The body is information-dense and domain-specific (no padding about what ClickHouse or an LLM is), but a few SQL queries are reused verbatim across sections and the Hog-vs-LLM comparison table is lengthy, leaving minor trim opportunities. Not a 5 because of that small redundancy; not a 3 because almost every line carries PostHog-specific load.	4 / 5
Actionability	Every workflow ships copy-paste-ready tool-call JSON and complete HogQL with exact property names ($ai_evaluation_result, $ai_evaluation_applicable, etc.), covering the common cases fully executably.	5 / 5
Workflow Clarity	Workflows are explicitly numbered (Step 1-4) and the LLM-judge build flow has a real feedback loop (create enabled:false → run → inspect → refine → enable). It is not a 5 because soft-delete/destroy paths lack an explicit confirmation checkpoint.	4 / 5
Progressive Disclosure	Section headers are clear and navigation is easy, but the skill is a ~410-line monolith with no bundle files — content like the full event-schema table and the SQL recipe library could live in one-level-deep reference files. The under-50-line simple-skill exception does not apply, so this stays at 3 rather than higher.	3 / 5
	Total	16 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: third-person voice, concrete actions, and an explicit trigger clause covering the realistic ways a user would invoke this skill. It cleanly satisfies the top anchor on every dimension.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions ('Find existing evaluations, inspect their configuration, run them against specific generations, query individual results, and set up scheduled reports') across three named evaluation types — comprehensive coverage matching the top anchor.	5 / 5
Completeness	Explicitly answers both 'what' (investigate/find/inspect/run/query/report) and 'when' (a full 'Use when the user asks to...' clause with concrete trigger phrases), matching the top anchor exactly.	5 / 5
Trigger Term Quality	The 'Use when...' clause packs natural trigger phrases a PostHog user would actually say ('debug why an evaluation is failing', 'dry-run a Hog evaluator', 'prototype a new LLM-judge prompt', 'inspect sentiment classifications'), giving comprehensive keyword coverage.	5 / 5
Distinctiveness Conflict Risk	A clear narrow niche — PostHog AI observability evaluations across three distinct typed variants (hog/llm_judge/sentiment) — with triggers unlikely to fire for unrelated skills, so conflict risk is minimal.	5 / 5
	Total	20 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: PostHog/posthog
Path: products/ai_observability/skills/exploring-llm-evaluations/SKILL.md
Commit: b0fece0

Reviewed: 3 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.