CtrlK
BlogDocsLog inGet started
Tessl Logo

creating-online-evaluations

Author continuously-running online evaluations in PostHog AI observability, grounded in a real failure mode you've identified. Use when the user wants an evaluation that automatically scores new generations or whole traces going forward — "create an eval to catch X", "continuously check that responses do Y", "turn this failure into an eval". Covers choosing the target and eval type (hog / llm_judge / sentiment), gating on the team's provider key before an llm_judge eval, scoping which generations trigger it via conditions (property filters + rollout sampling), creating it disabled, verifying scope, and enabling. Finding and ranking the failure modes worth evaluating is its own job — use exploring-ai-failures first. To debug or manage evaluations that already exist, use exploring-llm-evaluations.

SKILL.md
Quality
Evals
Security

No security issues found

Scanned

Repository
PostHog/posthog
Audited
Security analysis
Snyk

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.