llm-as-judge-evaluation

Evaluate LLM outputs using frontier models as judges. Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines. Covers position bias mitigation, statistical significance, and generating preference data for DPO/RLHF.

Quality

73%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./backend/cli/skills/llm-tools/llm-as-judge-evaluation/SKILL.md

Quality

Content

53%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A content-rich, actionable skill with strong executable code and a clear end-to-end checklist, undermined by duplicated swap-position logic, a missing validation feedback loop for its batch/deployment-gate workflow (capping workflow_clarity at 3), and reference files that exist but are never linked while their content is inlined.

Suggestions

Link the existing reference files from the body (e.g., 'See [pairwise-comparison.md](references/pairwise-comparison.md) for CoT judging, tie-rate interpretation, and pitfalls' and 'See [scoring-rubrics.md](references/scoring-rubrics.md) for rubric templates') and move the duplicated rubric-template and pairwise-pitfall content out of SKILL.md to fix progressive_disclosure.

Remove the mitigated_pairwise duplicate and consolidate position-bias mitigation into pairwise_compare, or make the wrapper a thin call that reuses the existing swap_positions path, to tighten conciseness.

Add an explicit validation feedback loop to the Quick Start Checklist — e.g., 'If the bootstrap CI straddles 0.5, the result is inconclusive: gather more examples and re-run', and a judge-API failure retry step — so the batch/deployment-gate workflow can score above 3.

Dimension	Reasoning	Score
Conciseness	Largely efficient and free of basic-concept padding, but the swap-position logic is duplicated between pairwise_compare(swap_positions=True) and the later mitigated_pairwise wrapper, and evaluate_model_pair re-does random position assignment — more than minor redundancy, matching the 'mostly efficient but some unnecessary content' anchor rather than 4.	3 / 5
Actionability	Provides fully executable Python (pairwise_compare, likert_score, bootstrap_win_rate, generate_dpo_pairs, multi_judge_compare) covering common cases, but likert_score references an undefined global client and multi_judge_compare's default judge list will date quickly, leaving minor gaps below the copy-paste-ready 5 anchor.	4 / 5
Workflow Clarity	The Quick Start Checklist gives a sequenced 7-step process with a significance check, but as a batch/deployment-gate skill it lacks an explicit validate→fix→retry feedback loop (e.g., judge API failures, CI straddling 0.5 triggering more data), so the batch-operations cap of 3 applies.	3 / 5
Progressive Disclosure	Two reference bundle files exist (references/pairwise-comparison.md, references/scoring-rubrics.md) but are never linked from the body, while their content is inlined — a monolithic structure where clearly-separable content stays inline and references are buried, matching the 'minimal structure, inlined content' anchor rather than the 3 anchor which requires clearly signaled references.	2 / 5
	Total	12 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that clearly answers both what it does and when to use it, with concrete trigger terms and a distinctive niche. Trigger-term synonyms could be marginally broader, but it is otherwise exemplary.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions — 'Evaluate LLM outputs using frontier models as judges', 'pairwise model comparison', 'quality scoring with custom rubrics', 'generating preference data for DPO/RLHF' — with comprehensive coverage, matching the anchor for several specific concrete actions.	5 / 5
Completeness	Explicitly answers both 'what' ('Evaluate LLM outputs using frontier models as judges') and 'when' ('Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines') with concrete trigger phrases, matching the top anchor.	5 / 5
Trigger Term Quality	Strong natural keyword coverage ('pairwise model comparison', 'quality scoring', 'custom rubrics', 'DPO/RLHF') but a few common synonyms (e.g., 'LLM evaluation', 'A/B testing models') are missing, sitting just below the comprehensive-synonyms anchor of 5.	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (LLM-as-judge evaluation) with distinctive triggers (position bias mitigation, DPO/RLHF preference data, bootstrap significance) that are unlikely to fire for unrelated skills, matching the minimal-conflict-risk anchor.	5 / 5
	Total	19 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/llm-tools/llm-as-judge-evaluation/SKILL.md
Commit: 3a6c3a9

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.