evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Quality

86%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

72%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill with real reference files and clear workflow checklists, but it is verbose for an overview and omits validation checkpoints in its batch evaluation workflows. Tightening generic code and adding result-verification steps would raise the two middle dimensions.

Suggestions

Trim generic Claude-known code (the matplotlib learning-curve plot and pandas comparison-table builder) or move them into a reference file; keep only the lm_eval-specific invocation patterns inline.

Add validation/verification checkpoints to the batch workflows: check each lm_eval exit code in eval_all_models.sh and confirm result JSON files exist and are non-empty before generating the comparison table.

Reduce repetition of the full lm_eval command block by defining the common --model_args once and showing only the differing flags per variation.

Dimension	Reasoning	Score
Conciseness	Mostly efficient with concrete commands, but includes generic code Claude already knows (matplotlib plotting, pandas DataFrame assembly) and repeats the same lm_eval invocation pattern many times; the ~480-line body could be tightened.	2 / 3
Actionability	Provides copy-paste-ready, fully executable bash commands and Python snippets throughout (e.g. the quick-start lm_eval invocation, eval_all_models.sh loop, vLLM config), matching the top anchor.	3 / 3
Workflow Clarity	Workflows are sequenced with checklists, but the batch model-comparison loop and checkpoint evaluation lack any validation/verification checkpoint (no exit-code checks, no confirmation that each eval succeeded before building the comparison table), capping this dimension at 2 per the batch-operations guideline.	2 / 3
Progressive Disclosure	The body is an overview with four well-signaled, one-level-deep references to real bundle files (benchmark-guide.md, custom-tasks.md, api-evaluation.md, distributed-eval.md), with advanced topics appropriately split out and easy navigation.	3 / 3
	Total	10 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description with an explicit 'Use when' trigger, concrete actions, named benchmarks, and a clear niche. It would benefit from trimming the marketing clause ('Industry standard used by EleutherAI, HuggingFace, and major labs.'), but every rubric dimension still meets the top anchor.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions ('Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag)', 'comparing models, reporting academic results, tracking training progress') with specific named benchmarks, matching the top anchor.	3 / 3
Completeness	Explicitly answers what ('Evaluates LLMs across 60+ academic benchmarks...') and when ('Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress.'), satisfying the top anchor for both.	3 / 3
Trigger Term Quality	Natural trigger phrases a user would say ('benchmarking model quality', 'comparing models', 'reporting academic results', 'tracking training progress') plus widely-known benchmark names give good coverage of real user terms.	3 / 3
Distinctiveness Conflict Risk	The LLM academic-benchmarking niche with named tools (lm-eval, EleutherAI, HuggingFace, vLLM) is clearly distinct and unlikely to trigger for unrelated skills.	3 / 3
	Total	12 / 12 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/lm-evaluation-harness/SKILL.md
Commit: 52845c3

Reviewed: about 9 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.