evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Quality

74%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./backend/cli/skills/ml-training/lm-evaluation-harness/SKILL.md

Quality

Content

61%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A thorough, code-rich skill body with well-structured workflows and properly signaled reference files. It is let down by verbosity from redundant checklists, a few non-executable bash inline comments, and the absence of validation checkpoints in batch evaluation workflows.

Suggestions

Remove the "Copy this checklist" blocks or the duplicated Step-header prose — keeping both restates the same sequence twice and inflates the body.

Fix bash commands that put a `#` comment after a `\` line-continuation (e.g. `--num_fewshot 5 \ # 5-shot`); move those comments onto their own line so the commands execute as written.

Add an explicit verification step to each workflow (e.g. after Step 3, confirm `results/*.json` exists and inspect stderr) so batch evaluations have validation checkpoints.

Dimension	Reasoning	Score
Conciseness	The body runs ~490 lines and includes padded sections — the "Copy this checklist" blocks restate the Step 1–4 headers that immediately follow, and vLLM commands are repeated between Workflow 4 and Common issues — so it is mostly efficient but could be tightened rather than lean.	3 / 5
Actionability	Extensive copy-paste-ready bash and python examples cover the common cases, but inline comments after backslash line-continuations (e.g. "--num_fewshot 5 \ # 5-shot evaluation") break those bash commands, leaving minor execution gaps that keep it below a 5.	4 / 5
Workflow Clarity	The four workflows have clear Step 1–4 sequences with checklists, but there are no validation/verification checkpoints (e.g. confirm the results JSON was written, check stderr) for batch evaluation runs, so validation gaps cap this at 3.	3 / 5
Progressive Disclosure	The body is an overview that points to four clearly-signaled one-level-deep references (benchmark-guide, custom-tasks, api-evaluation, distributed-eval), all of which exist, with most content appropriately placed; minor organization gaps keep it just below a 5.	4 / 5
	Total	14 / 20 Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that concretely names the domain, specific benchmarks, and explicit trigger conditions. It clearly answers both what the skill does and when to use it with minimal conflict risk.

Dimension	Reasoning	Score
Specificity	"Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag)" names the domain and several concrete benchmarks, with the trigger clause listing benchmarking/comparing/reporting/tracking actions — comprehensive but with minor gaps, so a 4 rather than a 5.	4 / 5
Completeness	It explicitly states what ("Evaluates LLMs across 60+ academic benchmarks") and when ("Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress") with concrete trigger phrases, matching the top anchor.	5 / 5
Trigger Term Quality	"benchmarking model quality, comparing models, reporting academic results, or tracking training progress" plus named benchmarks are natural phrases users would say; coverage is good but a few common variations (e.g. "eval", "test my model") are missing, so a 4 rather than a 5.	4 / 5
Distinctiveness Conflict Risk	The named harness plus the specific benchmark set (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag) carve a clear niche with distinct triggers and minimal overlap with other skills.	5 / 5
	Total	18 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/lm-evaluation-harness/SKILL.md
Commit: fcd2e11

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.