evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Quality

87%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

80%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable, well-structured content with executable examples and clean progressive disclosure to real reference files. The main gap is the batch model-evaluation workflow lacking validation/error-handling checkpoints, which caps workflow clarity.

Suggestions

Add error handling to the batch model-evaluation script (e.g., `set -e`, checking lm_eval exit codes, logging failures) so a failed run is caught before moving to the next model.

Add a verify step after batch evaluation confirming each results/$model_name.json was written and contains expected task metrics before generating the comparison table.

De-duplicate the recurring full lm_eval command blocks by referencing a base invocation and showing only the differing flags per workflow variant.

Dimension	Reasoning	Score
Conciseness	Mostly lean executable command/code examples with only brief, useful benchmark parentheticals rather than concept explanation; minor trimming possible where full lm_eval commands recur near-verbatim across workflows.	4 / 5
Actionability	Fully executable, copy-paste-ready commands and scripts covering HuggingFace, vLLM, quantized, custom-checkpoint, and multi-model comparison cases with concrete flags and expected output.	5 / 5
Workflow Clarity	Workflows are sequenced with checklists, but the batch model-comparison workflow (Workflow 3) loops over models with no validation or verification of eval success, so workflow clarity is capped at 3 per the batch-operation rule.	3 / 5
Progressive Disclosure	Clear overview body with well-signaled, one-level-deep references to four real files (benchmark-guide.md, custom-tasks.md, api-evaluation.md, distributed-eval.md), with bulk detail appropriately split out.	5 / 5
	Total	17 / 20 Passed

Description

95%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that clearly states capability and trigger conditions with concrete benchmark names. Slightly less than maximal specificity only because the listed actions are facets of one evaluation activity rather than genuinely distinct operations.

Dimension	Reasoning	Score
Specificity	Lists several concrete actions ('Evaluates LLMs across 60+ academic benchmarks', 'comparing models', 'reporting academic results', 'tracking training progress') plus named benchmarks, but the verbs are variations of one evaluation activity rather than fully distinct operations, leaving minor coverage gaps.	4 / 5
Completeness	Explicitly answers both what ('Evaluates LLMs across 60+ academic benchmarks...') and when ('Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Comprehensive natural terms users would say — 'benchmarking model quality', 'comparing models', 'reporting academic results', 'tracking training progress' — plus the benchmark names (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) that practitioners actually invoke.	5 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (academic LLM benchmarking) with distinct triggers and specific benchmark names, making conflict with unrelated skills unlikely.	5 / 5
	Total	19 / 20 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
metadata_version	'metadata.version' is missing	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	14 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/lm-evaluation-harness/SKILL.md
Commit: 543b17f

Reviewed: 2 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.