CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

evaluation-lm-evaluation-harness

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.

Quality

82%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

71%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill with executable commands for every major workflow and excellent reference-file separation for advanced topics. Its weaknesses are token efficiency (duplicated checklist blocks and glosses of known benchmarks) and small correctness slips in a few code examples.

Suggestions

Remove the 'Copy this checklist' blocks — their step names are already duplicated as the bolded step headings that follow, so they add ~20 lines of pure redundancy.

Fix the broken bash line-continuation comments (e.g. '--num_fewshot 5 \ # 5-shot evaluation (standard)'): move comments to their own line or drop the trailing backslash-comment pattern so commands are truly copy-paste executable.

Align the Workflow 2 plotting script with the tasks it actually evaluates (it reads data["results"]["mmlu"] but the checkpoint eval uses gsm8k,hellaswag) and add the missing 'import glob', or trim the one-line benchmark descriptions Claude already knows.

DimensionReasoningScore

Conciseness

The four 'Copy this checklist' blocks (~20 lines) duplicate the step headings that immediately follow, and one-line glosses like 'GSM8K - Grade school math word problems' explain benchmarks Claude already knows; beyond these the content is command-dense, so it is mostly efficient but needs tightening rather than showing only minor trimmable instances (anchor 4).

3 / 5

Actionability

Copy-paste-ready lm_eval commands and complete bash/python scripts cover the common cases (HF, quantized, custom checkpoint, vLLM, multi-model), but a few executability gaps remain: inline '# comment' placed after line-continuation backslashes breaks those bash commands, the Workflow 2 plot script reads data["results"]["mmlu"] though that workflow evaluated gsm8k/hellaswag, and glob is used without import.

4 / 5

Workflow Clarity

All four workflows have clearly sequenced numbered steps plus a troubleshooting 'Common issues' section, but validation checkpoints are mostly implicit ('Analyze results') rather than explicit validate-fix-retry loops, keeping it below anchor 5.

4 / 5

Progressive Disclosure

The body is a clear overview (quick start, common workflows, troubleshooting) with advanced material — benchmark-guide, custom-tasks, api-evaluation, distributed-eval — split into four one-level-deep reference files that all exist and are well signaled with descriptive link text, matching anchor 5.

5 / 5

Total

16

/

20

Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it states concrete capabilities with named benchmarks and backends, uses third person, and includes an explicit 'Use when...' clause with natural trigger phrases. The only minor gap is that it describes a single primary action (evaluation) rather than a fuller range of distinct actions.

DimensionReasoningScore

Specificity

Concrete specifics are present — 'Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag)' and 'Supports HuggingFace, vLLM, APIs' — but the action set is essentially one (evaluate) rather than the multiple distinct concrete actions of anchor 5, while exceeding anchor 3's 1-2 actions.

4 / 5

Completeness

Clearly answers 'what' ('Evaluates LLMs across 60+ academic benchmarks...', 'Supports HuggingFace, vLLM, APIs') and 'when' with an explicit 'Use when...' clause containing concrete trigger phrases, matching anchor 5; anchor 4's weaker/implied 'when' does not apply.

5 / 5

Trigger Term Quality

'benchmarking model quality, comparing models, reporting academic results, or tracking training progress' plus benchmark names like MMLU and HumanEval are natural user phrases; common variations such as 'run evals' or 'evaluate my model' are missing, so it falls just short of anchor 5's comprehensive synonym coverage.

4 / 5

Distinctiveness Conflict Risk

The niche (standardized academic LLM benchmarking with named benchmarks and the EleutherAI/HuggingFace ecosystem) is distinct with clear triggers, giving minimal conflict risk with other skills.

5 / 5

Total

18

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

metadata_field

'metadata' should map string keys to string values

Warning

Total

15

/

16

Passed

Repository

davila7/claude-code-templates

Path

cli-tool/components/skills/ai-research/evaluation-lm-evaluation-harness/SKILL.md

Revision

cc3f535

Reviewed

Table of Contents

DiscoveryImplementationValidation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.

Claim skill