Exploring AI observability evaluations

PostHog evaluations score $ai_generation events. Each evaluation is one of three types:

• hog — deterministic Hog code that returns true/false (and optionally N/A). Best for objective rule-based checks: format validation (JSON parses, schema matches), length limits, keyword presence/absence, regex patterns, structural assertions, latency thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this when the criterion can be expressed as code.
• llm_judge — an LLM scores generations against a prompt you write. Best for subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic drift, instruction-following. Costs an LLM call per run and requires AI data processing approval at the org level.
• sentiment — classifies sentiment from user messages on each matching generation. Returns a sentiment label and score, not a pass/fail verdict.

Results from all types land in ClickHouse as $ai_evaluation events. Boolean evaluations (llm_judge and hog) set $ai_evaluation_result; sentiment evaluations set $ai_sentiment_* properties instead.

This skill covers the full lifecycle: list/inspect/manage evaluation configs, run them on specific generations, query individual results, and configure evaluation reports that summarize recent runs on a schedule.

Tools

All llma-evaluation-* tools are defined in products/ai_observability/mcp/tools.yaml.

Event schema

Every run of an evaluation emits an $ai_evaluation event. Key properties:

When $ai_evaluation_applicable = false, the run counts as N/A regardless of $ai_evaluation_result. For evaluations that don't support N/A, this property may be null — treat null as "applicable".

Workflow: investigate why an evaluation is failing

Works the same way for boolean llm_judge and hog evaluations — the differences only matter when you eventually go to fix the evaluator (edit the prompt vs. edit the Hog source). Sentiment evaluations should be inspected by sentiment label and score rather than pass/fail filters.

Step 1 — Find the evaluation

posthog:llma-evaluation-list
{ "search": "hallucination", "enabled": true }

Look at the returned id, name, evaluation_type, and either:

• evaluation_config.prompt for an llm_judge
• evaluation_config.source for a hog evaluator

The Hog source is the ground truth for why a hog evaluator passes or fails — read it before assuming the failure is in the generation.

Step 2 — Break down pass, fail, and N/A

posthog:execute-sql
SELECT
    countIf(properties.$ai_evaluation_applicable = false) AS na_count,
    countIf(
        (properties.$ai_evaluation_applicable IS NULL
            OR properties.$ai_evaluation_applicable != false)
        AND properties.$ai_evaluation_result = true
    ) AS pass_count,
    countIf(
        (properties.$ai_evaluation_applicable IS NULL
            OR properties.$ai_evaluation_applicable != false)
        AND properties.$ai_evaluation_result = false
    ) AS fail_count
FROM events
WHERE event = '$ai_evaluation'
    AND properties.$ai_evaluation_id = '<evaluation_uuid>'
    AND timestamp >= now() - INTERVAL 7 DAY

If the evaluation already has report configs, llma-evaluation-report-list and llma-evaluation-report-run-list give you the AI-written reports from earlier periods, which is a fast way to see how the picture has moved.

Step 3 — Read the failing runs

The reasoning text is where the pattern shows up. Pull the recent fails and read them:

posthog:execute-sql
SELECT
    properties.$ai_target_event_id AS generation_id,
    properties.$ai_trace_id AS trace_id,
    properties.$ai_evaluation_reasoning AS reasoning,
    timestamp
FROM events
WHERE event = '$ai_evaluation'
    AND properties.$ai_evaluation_id = '<evaluation_uuid>'
    AND properties.$ai_evaluation_result = false
    AND (
        properties.$ai_evaluation_applicable IS NULL
        OR properties.$ai_evaluation_applicable != false
    )
    AND timestamp >= now() - INTERVAL 7 DAY
ORDER BY timestamp DESC
LIMIT 25

The N/A guard (IS NULL OR != false) is important — it matches the same logic the backend uses to bucket runs.

Step 4 — Drill into example failing runs

Take the most representative rows from Step 3 and pull the underlying trace:

posthog:query-llm-trace
{ "traceId": "<trace_id>", "dateRange": {"date_from": "-30d"} }

(If you only have a generation ID, query for it via execute-sql first to find the parent trace ID.)

Workflow: run an evaluation against a specific generation

Use this when the user pastes a trace/generation URL and asks "what would evaluation X say about this?".

posthog:llma-evaluation-run
{
  "evaluationId": "<eval_uuid>",
  "target_event_id": "<generation_event_uuid>",
  "timestamp": "2026-04-01T19:39:20Z",
  "event": "$ai_generation"
}

The timestamp is required for an efficient ClickHouse lookup of the target event. Pass distinct_id if you have it — it speeds up the lookup further.

Workflow: build and test a new evaluator

Hog evaluator (deterministic, code-based)

Reach for this first when the criterion is rule-based — it's cheaper, faster, and reproducible. Prototype with llma-evaluation-test-hog (no save):

posthog:llma-evaluation-test-hog
{
  "source": "return event.properties.$ai_output_choices[1].content contains 'sorry';",
  "sample_count": 5,
  "allows_na": false
}

The handler returns the boolean result for each of the most recent N $ai_generation events. Iterate on the source until it behaves as expected, then promote it via llma-evaluation-create:

posthog:llma-evaluation-create
{
  "name": "Output is valid JSON",
  "description": "Fails when the assistant message can't be parsed as JSON",
  "evaluation_type": "hog",
  "evaluation_config": {
    "source": "let raw := event.properties.$ai_output_choices[1].content; try { jsonParseStr(raw); return true; } catch { return false; }"
  },
  "output_type": "boolean",
  "enabled": true
}

Hog evaluators have full access to the event and its properties — common patterns include schema validation, length/token limits, regex matches, and tool-call shape checks. Because they're deterministic, results are reproducible across reruns and trivially diff-able.

LLM-judge evaluator (subjective, prompt-based)

Use this when the criterion is fuzzy and a code rule would be brittle (tone, factuality, helpfulness, on-topic-ness). There's no equivalent of llma-evaluation-test-hog for LLM judges — the typical loop is to create the evaluator with enabled: false, run it manually against a handful of representative generations via llma-evaluation-run, inspect the results, refine the prompt with llma-evaluation-update, and then flip enabled: true when you're satisfied:

posthog:llma-evaluation-create
{
  "name": "Response stays on-topic",
  "description": "LLM judge — fails if the assistant changes topic from the user's question",
  "evaluation_type": "llm_judge",
  "evaluation_config": {
    "prompt": "You are evaluating whether the assistant's reply stays on-topic relative to the user's most recent question. Return true if it does, false if the assistant changed the subject. Return N/A if the user did not actually ask a question."
  },
  "output_type": "boolean",
  "output_config": { "allows_na": true },
  "model_configuration": {
    "provider": "openai",
    "model": "gpt-5-mini"
  },
  "enabled": false
}

Then dry-run against a known-good and a known-bad generation:

posthog:llma-evaluation-run
{
  "evaluationId": "<new_eval_uuid>",
  "target_event_id": "<generation_uuid>",
  "timestamp": "2026-04-01T19:39:20Z"
}

LLM judges require organisation AI data processing approval. Hog evaluators do not.

Workflow: manage the evaluation lifecycle

llm_judge evaluations require AI data processing approval at the org level (is_ai_data_processing_approved). Hog evaluations do not require this gate — they run as plain code on the ingestion pipeline.

When to use Hog vs LLM judge

Reach for Hog by default. Switch to LLM judge only when the criterion can't be expressed as code.

A common pattern is to layer them: a Hog evaluator gates obvious format/length violations cheaply, and an LLM-judge evaluator only fires on the generations that pass the Hog gate (via conditions).

Investigation patterns

Diagnosis works the same way regardless of whether the evaluator is hog or llm_judge — you read the resulting $ai_evaluation events, not the evaluator itself. The fix path differs (edit Hog source vs. edit prompt) but the diagnosis is identical.

"Why is evaluation X suddenly failing more?"

1. llma-evaluation-list — confirm the evaluation is still enabled and unchanged (compare evaluation_config.source or evaluation_config.prompt to the version you expect)

2. Read the recent failing runs and their reasoning (Step 3 above) and group them into the dominant failure patterns

3. SQL count of fails per day to confirm the regression window:

   SELECT toDate(timestamp) AS day, count() AS fails
   FROM events
   WHERE event = '$ai_evaluation'
       AND properties.$ai_evaluation_id = '<uuid>'
       AND properties.$ai_evaluation_result = false
       AND timestamp >= now() - INTERVAL 30 DAY
   GROUP BY day
   ORDER BY day

4. Drill into a representative trace per pattern via query-llm-trace

"Are passes and fails caused by the same root content?"

1. Pull two samples with the Step 3 query, flipping $ai_evaluation_result between true and false
2. If the passing and failing runs describe similar content:
   • For an llm_judge: the prompt or rubric is probably ambiguous — reword evaluation_config.prompt and use llma-evaluation-update
   • For a hog evaluator: the rule is probably under- or over-matching — read the source via llma-evaluation-get, narrow the predicate, and retest with llma-evaluation-test-hog before pushing the fix via llma-evaluation-update

"Did a Hog evaluator regression after a code change?"

Hog evaluators are reproducible — if the source hasn't changed, identical inputs should yield identical outputs. When fail rates jump for a Hog evaluator:

1. llma-evaluation-get — note the current source and updated_at
2. Spot-check the latest failing runs with the SQL query from Step 4 above
3. Re-run the source against those exact generations using llma-evaluation-test-hog with a modified conditions filter that targets them
4. If the test results match the live results, the change is in the generations, not the evaluator (a model upgrade, prompt change upstream, etc.) — investigate the producer
5. If they diverge, the evaluator was edited; check git history of the source field via the activity log

"What kinds of generations does this evaluator skip as N/A?"

posthog:execute-sql
SELECT
    properties.$ai_target_event_id AS generation_id,
    properties.$ai_trace_id AS trace_id,
    properties.$ai_evaluation_reasoning AS reasoning,
    timestamp
FROM events
WHERE event = '$ai_evaluation'
    AND properties.$ai_evaluation_id = '<evaluation_uuid>'
    AND properties.$ai_evaluation_applicable = false
    AND timestamp >= now() - INTERVAL 7 DAY
ORDER BY timestamp DESC
LIMIT 25

Read the reasoning on those runs to see whether the N/A logic is doing the right thing. If a run looks like something that should have been scored:

• For an llm_judge: the applicability instruction in the prompt is too broad — narrow it
• For a hog evaluator with output_config.allows_na: true: the source is returning null (or whatever the N/A signal is) too eagerly — tighten the precondition

"Score this single generation right now"

llma-evaluation-run with the trace's generation ID and timestamp. Useful for spot-checking or wiring evaluations into a larger agent loop.

Constructing UI links

• Evaluations list: https://app.posthog.com/ai-evals/evaluations
• Single evaluation: https://app.posthog.com/ai-evals/evaluations/<evaluation_id>
• Underlying generation/trace: see the exploring-llm-traces skill's URL conventions

Always surface the relevant link so the user can verify in the UI.

Tips

• Evaluation reports are configured per evaluation with llma-evaluation-report-create: frequency: "scheduled" with an rrule for a daily or weekly cadence, or frequency: "every_n" with a trigger_threshold to fire once that many new results have accumulated. Delivery goes to email or Slack via delivery_targets
• llma-evaluation-report-generate runs a configured report immediately instead of waiting for the next trigger; llma-evaluation-report-run-list returns past runs with the report content they delivered
• For rich filtering not supported by llma-evaluation-list (e.g. by author or model configuration), fall back to execute-sql against the evaluations Postgres table or the $ai_evaluation ClickHouse events
• When showing failure patterns to the user, always include 1-2 example trace links so they can validate the pattern visually
• llma-evaluation-* tools use evaluation:read for read tools and evaluation:write for mutating tools; the llma-evaluation-report-* tools use llm_analytics:read and llm_analytics:write
• Hog evaluators are reproducible — if you suspect a regression, llma-evaluation-test-hog with the suspect source against the failing generations is the fastest way to bisect whether the change is in the evaluator or in the producer of the generations
• LLM-judge evaluators are non-deterministic across reruns; expect 1-5% noise even with a fixed prompt and model. If you're chasing a small regression in fail rate, prefer Hog or pin a deterministic provider/seed in the model_configuration

Related skills

• creating-online-evaluations — author a new evaluation from scratch
• exploring-ai-failures — ground the next evaluation in observed failure modes