evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Quality

80%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./bundled/skills/evaluating-llms-harness/SKILL.md

Quality

Content

72%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable content with excellent progressive disclosure and copy-paste-ready examples across the main workflows. The main weaknesses are redundant checklists/benchmark glosses that pad the token budget and missing validation checkpoints in the batch evaluation workflows.

Suggestions

Remove the per-workflow 'Copy this checklist' code blocks; the bold Step 1-4 headers already sequence the work and the duplicate checklists waste tokens.

Drop or condense the one-line benchmark descriptions (e.g. 'GSM8K - Grade school math word problems') since Claude already knows these; keep only non-obvious details like runtime or shot-count conventions.

Add explicit validation checkpoints to the batch workflows, e.g. after eval_all_models.sh verify each results/*.json exists and check for empty result blocks before generating the comparison table.

Dimension	Reasoning	Score
Conciseness	The body is heavily code/commands (efficient), but includes redundant per-step checklists that duplicate the step headers and one-line benchmark descriptions Claude already knows (e.g. 'GSM8K - Grade school math word problems'), so it is mostly efficient but could be tightened.	3 / 5
Actionability	Fully executable, copy-paste-ready commands and scripts cover the common cases (HF, vLLM, quantized, custom checkpoint, multi-model comparison, training tracking) with concrete flags and example output.	5 / 5
Workflow Clarity	Four workflows are clearly sequenced with Step 1-4 checklists, but batch/periodic operations (eval_all_models.sh, checkpoint eval) lack validation checkpoints (e.g. verify output JSON, detect failed runs), which caps this dimension at 3.	3 / 5
Progressive Disclosure	SKILL.md is a clear overview with well-signaled, one-level-deep references to real files (benchmark-guide.md, custom-tasks.md, api-evaluation.md, distributed-eval.md), with advanced content appropriately split out and easy to navigate.	5 / 5
	Total	16 / 20 Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that clearly states what the skill does and when to use it, with concrete benchmark names and natural trigger phrases. Minor gaps in action variety and synonym coverage keep specificity and trigger_term_quality just below maximum.

Dimension	Reasoning	Score
Specificity	Lists concrete actions ('Evaluates LLMs across 60+ academic benchmarks', 'Supports HuggingFace, vLLM, APIs') with five named benchmarks, but the core action set is essentially evaluate + support backends rather than many distinct verbs, so it sits just below the comprehensive 5 anchor.	4 / 5
Completeness	Explicitly answers both what ('Evaluates LLMs across 60+ academic benchmarks...') and when ('Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Natural phrases users would say are present ('benchmarking model quality, comparing models, reporting academic results, tracking training progress') plus named benchmarks, but a few common variations ('eval', 'leaderboard', 'test a model') are missing.	4 / 5
Distinctiveness Conflict Risk	Clear niche (academic LLM benchmarking harness) with distinct triggers and named benchmarks; minimal overlap risk with other skills.	5 / 5
	Total	18 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: foryourhealth111-pixel/Vibe-Skills
Path: bundled/skills/evaluating-llms-harness/SKILL.md
Commit: d5ae560

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.