CtrlK
BlogDocsLog inGet started
Tessl Logo

creating-online-evaluations

Author continuously-running online evaluations in PostHog AI observability, grounded in real failure modes you've identified. Use when the user wants evaluations that automatically score new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn these failures into evals". Covers letting the explored data decide how many evals to create, proposing that set in plain language and asking the user which ones they want, choosing the target and eval type (hog / llm_judge / sentiment), configuring a provider, model, and usable provider key for an llm_judge eval, scoping which generations trigger it via conditions, creating disabled, verifying scope, and enabling. Falls back to proposing a sentiment eval when no failure mode is worth catching. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.

76

Quality

96%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Loading evals

Repository
PostHog/posthog

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.