Name: writing-evals
Rating: 69.6 (1 reviews)
Author: PostHog

writing-evals

Teaches how to write and run evals on the `products/posthog_ai/eval_harness/` harness — sandboxed agent suites that execute the real coding agent in a Docker or Modal sandbox against a seeded Hedgebox project, and one-shot suites that score a single in-process model invocation per case. Use when adding or changing eval suites, cases, scorers, seeders, or synthesizers under `products/posthog_ai/evals/` or `products/*/evals/`, when touching the harness under `products/posthog_ai/eval_harness/`, or when running or debugging those evals (`hogli evals`). Covers suite kinds and discovery, case anatomy, the seeder/synthesizer split, the one-branch scorer patterns, and how to read results. Not for `ee/hogai/eval/ci/` pytest evals, and not for the LLM Analytics product's evaluation features.

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured authoring guide that is actionable, assumes Claude's competence, and cleanly splits detail into one-level-deep references. The main weaknesses are minor verbosity in a few rationale passages and the absence of an explicit validate-retry loop in the primary writing workflow.

Suggestions

Add an explicit validate-fix-retry feedback loop to the 'Writing a suite' workflow (e.g. run the case alone, read the transcript, fix, re-run) mirroring the verification checklist so the authoring path itself has checkpoints.

Tighten the suite-splitting and synthesizer-split rationale paragraphs to the governing rule plus one example, trimming the explanatory padding.

Fill in the '...' elisions in the deterministic scorer and one-shot task examples (e.g. a complete Score return and a concrete output dict) so the snippets are fully copy-paste ready.

Dimension	Reasoning	Score
Conciseness	Information-dense throughout — it assumes Claude knows Python/async/Docker and never explains basics — but a few paragraphs (e.g. the suite-splitting rationale, the synthesizer split) could be tightened without losing the actionable rule.	4 / 5
Actionability	Provides concrete, mostly copy-paste-ready code for both suite kinds and both scorer patterns, plus exact import paths, field names, and CLI commands; minor gaps appear as '...' elisions in the scorer and task examples.	4 / 5
Workflow Clarity	The authoring flow is sectioned clearly and the 'Verification checklist' gives a sequenced run-read-debug loop with checkpoints, but the main writing workflow lacks an explicit validate-and-retry feedback loop of its own.	4 / 5
Progressive Disclosure	SKILL.md stays an overview and delegates field-level API detail to references/authoring-reference.md and run setup to references/running-evals.md (both real files), with each reference clearly signaled by what it contains and kept one level deep.	5 / 5
	Total	17 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that clearly states what the skill does, gives explicit 'Use when' triggers, and carves out exclusions to avoid conflict with related eval tooling. Trigger-term coverage is the only minor gap, lacking synonyms or file extensions.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions — 'write and run evals', 'execute the real coding agent... against a seeded Hedgebox project', 'score a single in-process model invocation per case', plus authoring actions (suites, cases, scorers, seeders, synthesizers) — comprehensive coverage of the skill's capabilities.	5 / 5
Completeness	Explicitly answers both what (teaches writing/running sandboxed and one-shot eval suites on the harness) and when (a concrete 'Use when adding or changing eval suites, cases, scorers, seeders, or synthesizers... when running or debugging those evals') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Good natural-term coverage ('evals', 'eval suites', 'scorers', 'seeders', 'synthesizers', 'harness', 'hogli evals') that users of this system would actually say, but lacks synonyms or file extensions to reach comprehensive coverage.	4 / 5
Distinctiveness Conflict Risk	Clear niche scoped to products/posthog_ai/eval_harness, and the 'Not for ee/hogai/eval/ci/ pytest evals, and not for the LLM Analytics product's evaluation features' exclusion explicitly disambiguates from adjacent eval systems.	5 / 5
	Total	19 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
relative_links	Relative link issues: 3 suspicious	Warning

	Total	15 / 16 Passed

Repository: PostHog/posthog
Path: .agents/skills/writing-evals/SKILL.md
Commit: 6fca5f8

Reviewed: 3 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.