Name: creating-online-evaluations
Rating: 76 (1 reviews)
Author: PostHog

creating-online-evaluations

Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified. Use when the user wants evaluations that automatically score new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn these failures into evals". Covers letting the explored data decide how many evals to create, proposing that set in plain language and asking the user which ones they want, choosing the target and eval type (hog / llm_judge / sentiment), configuring a provider, model, and usable provider key for an llm_judge eval, scoping which generations trigger it via conditions, creating disabled, verifying scope, and enabling. Falls back to proposing a sentiment eval when no failure mode is worth catching. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.

Quality

96%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

92%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, actionable skill body that sequences a multi-step, cost-sensitive workflow with explicit validation, keeps the dense payload reference one level deep, and assumes Claude's competence throughout. The only weakness is minor verbosity in a few edge-case explanations.

Suggestions

Tighten the session-evaluation skip semantics (2.2) and the trace-ID reuse caveat (2.5) to a line or two each; the core constraints are what matter and the long form risks being skimmed.

Consider moving the detailed session/trace settle-config bounds tables (2.2) into references/evaluation-payload.md, keeping only the decision guidance and defaults inline.

A few Tips entries restate guidance already in the Phases; dedupe so the Tips section adds only what the workflow body does not already cover.

Dimension	Reasoning	Score
Conciseness	Mostly lean and assumes competence (no generic explanation of PostHog, evals, or Hog), but sections like the session skip semantics and trace-ID reuse caveats, while domain-relevant, run a touch long and could be tightened. Efficient with only minor over-explanation.	4 / 5
Actionability	Provides concrete tool names, a copy-paste ready create-payload JSON, exact field shapes, executable verification SQL, and specific generate-app-url calls — fully executable guidance covering the common hog and llm_judge cases.	5 / 5
Workflow Clarity	Clear phased sequence (1.1–1.3 decide, 2.1–2.6 build) with an explicit validation checkpoint in 2.5 (verify scope with SQL before enabling), a fix-and-retry feedback loop, and a hard guardrail to keep all evals disabled until the set is verified — well suited to the cost-bearing enable step.	5 / 5
Progressive Disclosure	SKILL.md is a clear overview with the bulk field/config reference offloaded to the real, one-level-deep references/evaluation-payload.md, signalled twice in the body; content is appropriately split and easy to navigate.	5 / 5
	Total	19 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that states concrete capabilities, gives explicit trigger phrases for when to use it, and cleanly separates its scope from sibling skills. It earns the top anchor on all four dimensions.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions across the full lifecycle — choosing eval type (hog/llm_judge/sentiment), configuring provider, model and usable provider key, scoping via conditions, creating disabled, verifying scope, and enabling — giving comprehensive coverage.	5 / 5
Completeness	Explicitly answers both what ("Author continuously-running online evaluations... grounded in real failure modes" plus the covered lifecycle steps) and when ("Use when the user wants evaluations that automatically score new generations or whole traces going forward") with concrete trigger phrases.	5 / 5
Trigger Term Quality	Includes natural phrasings a user would actually say such as "create an eval to catch X", "continuously check that responses do Y", and "turn these failures into evals", covering the common ways this need is voiced.	5 / 5
Distinctiveness Conflict Risk	Occupies a clear niche — creating new online evals — and draws explicit boundaries by routing failure-mode discovery to `exploring-ai-failures` and debug/management to `exploring-llm-evaluations`, minimizing overlap risk.	5 / 5
	Total	20 / 20 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
relative_links	Relative link issues: 2 missing	Warning
referenced_paths_exist	Referenced path issues: 4 missing	Warning

	Total	14 / 16 Passed

Repository: PostHog/posthog
Path: products/ai_observability/skills/creating-online-evaluations/SKILL.md
Commit: 6fca5f8

Reviewed: 3 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.