evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Quality

83%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable content with good progressive disclosure via real reference files, but it is somewhat verbose and its batch workflows omit the validation/verification checkpoints the rubric requires.

Suggestions

Add validation steps to the batch workflows: check each model's result file exists and eval exit code before generating the comparison table, and verify checkpoint eval succeeded before plotting learning curves.

Tighten conciseness by consolidating the repeated lm_eval invocations into a single parameterized example and moving the full JSON results block and matplotlib plotting script into a reference file.

Reduce duplicated boilerplate across Workflows 1-4 by referencing a shared 'common flags' snippet instead of restating --model, --device, and --batch_size in every block.

Dimension	Reasoning	Score
Conciseness	The body is mostly efficient and avoids patronizing concept explanations, but it repeats near-identical lm_eval invocations across workflows and inlines full JSON results, comparison tables, and plotting code that could be tightened or moved to references.	3 / 5
Actionability	Provides fully executable, copy-paste-ready bash and python blocks with real model names, flags, and example outputs covering the common benchmarking cases.	5 / 5
Workflow Clarity	Workflows 1-4 are clearly numbered and sequenced with checklists, but batch operations (evaluating all models, periodic checkpoint evaluation) lack validation/verification checkpoints (e.g. checking eval exit status or result-file existence before building tables), which caps the score per the batch-operations guideline.	3 / 5
Progressive Disclosure	SKILL.md is a concise overview with well-signaled, one-level-deep markdown links to four existing reference files (benchmark-guide, custom-tasks, api-evaluation, distributed-eval), making navigation easy.	5 / 5
	Total	16 / 20 Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, concrete description that covers both capabilities and clear trigger conditions with named benchmarks and frameworks. Slight room to broaden the action vocabulary and trigger synonyms beyond the four listed.

Dimension	Reasoning	Score
Specificity	Names the domain and enumerates five concrete named benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag), though the action set is essentially a single verb ('Evaluates') rather than multiple distinct actions, leaving minor coverage gaps.	4 / 5
Completeness	Explicitly answers both what ('Evaluates LLMs across 60+ academic benchmarks...') and when ('Use when benchmarking model quality, comparing models...') with concrete trigger phrases.	5 / 5
Trigger Term Quality	'benchmarking model quality, comparing models, reporting academic results, tracking training progress' are natural phrases users would say, with good coverage but a few synonyms (e.g. 'scoring', 'testing models') missing.	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (academic LLM benchmarking) named with specific frameworks (lm-eval, EleutherAI, HuggingFace, vLLM), making overlap with other skills minimal.	5 / 5
	Total	18 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: OpenRaiser/NanoResearch
Path: skills/vendor-ai-research/lm-evaluation-harness/SKILL.md
Commit: 9d3b440

Reviewed: ~21 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.