CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

llm-as-judge-evaluation

Evaluate LLM outputs using frontier models as judges. Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines. Covers position bias mitigation, statistical significance, and generating preference data for DPO/RLHF.

69

Quality

87%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable skill body with executable code throughout and a clear end-to-end workflow, held back by inlined reference material and orphaned bundle files. The main structural fix is linking the existing references/ files and moving bulk templates out of the main body.

Suggestions

Link the existing references/pairwise-comparison.md and references/scoring-rubrics.md from the body (e.g., under 'Advanced' sections) so the bundle files are discoverable, and move inlined bulk content like the full CODE_RUBRIC template into references/scoring-rubrics.md.

Remove the duplicated 'Position Bias Mitigation' section (mitigated_pairwise re-implements logic already inside pairwise_compare's swap_positions) or consolidate into one canonical implementation.

Add error-recovery guidance to the Quick Start Checklist for common failure modes: judge JSON parse errors, API failures/retries, and handling high tie rates before making deployment decisions.

DimensionReasoningScore

Conciseness

The body is code-dense and avoids explaining concepts Claude already knows, but contains redundancy: the 'Position Bias Mitigation' section re-implements swap logic already built into pairwise_compare, and the full 30-line CODE_RUBRIC example pads the rubric-design section. Fits anchor 4 (efficient, minor instances that could be trimmed) rather than 5 (every token earns its place).

4 / 5

Actionability

All guidance is fully executable, copy-paste-ready Python covering the common cases: pairwise comparison with position swapping, Likert scoring, bootstrap confidence intervals, DPO pair generation, and multi-judge ensembles. Matches anchor 5 (fully executable code with specific examples covering common cases).

5 / 5

Workflow Clarity

The 'Quick Start Checklist' provides a clear 7-step sequence with checkpoints (define criteria, check significance via bootstrap CI, decision gate), matching anchor 4 (clear sequence, most checkpoints present). Not 5 because there are no error-recovery/feedback loops for failure modes the code itself will hit — API errors, malformed judge JSON, or empty results after ties.

4 / 5

Progressive Disclosure

Two reference files exist (references/pairwise-comparison.md, references/scoring-rubrics.md) but the body never links to or signals them, and reference-type bulk content (full rubric templates, chain-of-thought judging) is inlined in SKILL.md. This matches anchor 3 (references present but not clearly signaled; content that should be separate is inline) rather than 4, where references would be mostly clearly signaled.

3 / 5

Total

16

/

20

Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that clearly states capabilities and an explicit 'Use for...' trigger clause with concrete, domain-specific actions. The only weakness is missing a few natural synonym phrases users might say when reaching for this skill.

DimensionReasoningScore

Specificity

The description lists multiple specific concrete actions — 'pairwise model comparison', 'quality scoring with custom rubrics', 'automated evaluation pipelines', 'generating preference data for DPO/RLHF' — with comprehensive coverage of the skill's capabilities, matching the anchor for multiple specific concrete actions. It is not 4 because there are no meaningful coverage gaps relative to what the skill body delivers.

5 / 5

Completeness

It explicitly answers both: what ('Evaluate LLM outputs using frontier models as judges') and when (explicit 'Use for pairwise model comparison, quality scoring with custom rubrics, and automated evaluation pipelines'). This matches the anchor for clearly and explicitly answering both what AND when with concrete trigger phrases; score 4 would require the 'when' to be less explicit.

5 / 5

Trigger Term Quality

Good keyword coverage with natural terms like 'LLM outputs', 'judges', 'pairwise model comparison', 'rubrics', 'DPO/RLHF', but a few natural phrases users would say are missing — notably 'LLM-as-judge' itself (only in name/tags), plus 'evals', 'benchmark', or 'win rate'. Fits anchor 4 (good coverage, a few natural terms missing) rather than 5 (comprehensive coverage including synonyms).

4 / 5

Distinctiveness Conflict Risk

The description carves out a clear niche (LLM evaluation via judge models) with distinct triggers (pairwise comparison, rubric scoring, DPO/RLHF preference data) that are unlikely to fire for unrelated skills. Minimal conflict risk; matches anchor 5.

5 / 5

Total

19

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

synthetic-sciences/openscience

Path

backend/cli/skills/llm-tools/llm-as-judge-evaluation/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.