exploring-llm-evaluations
Investigate AI observability evaluations — `hog` (deterministic code-based), `llm_judge` (LLM-prompt-based), and `sentiment` (user-message sentiment). Find existing evaluations, inspect their configuration, run them against specific generations, query individual results, and generate AI-powered summaries for boolean pass/fail runs. Use when the user asks to debug why an evaluation is failing, surface common failure modes, compare results across filters, dry-run a Hog evaluator, prototype a new LLM-judge prompt, inspect sentiment classifications, or manage the evaluation lifecycle.
75
92%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
View guide
Low
Low-risk findings worth noting
Low
Low-risk findings.
1 low severity finding. Worth noting, but not necessarily harmful.
W011: Third-party content exposure detected (indirect prompt injection risk).
What this means
The skill exposes the agent to untrusted, user-generated content from public third-party sources, creating a risk of indirect prompt injection. This includes browsing arbitrary URLs, reading social media posts or forum comments, and analyzing content from unknown websites.
Why it was flagged
The skill ingests outsider-authored free text at runtime when it reads `$ai_evaluation_reasoning` (LLM judge output) and, more importantly, the underlying `$ai_generation` trace content via `posthog:query-llm-trace`/`posthog:execute-sql` to investigate failures or run evaluations against specified generations.
- Repository
- PostHog/posthog
- Commit
6fca5f8
- Audited
- Security analysis
Is this your skill?
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.