CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

evaluating-llms-harness

Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs. Also covers NeMo Evaluator (guides in references/).

Quality

77%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./skills/evaluation/lm-evaluation-harness/SKILL.md
SKILL.md
Quality
Evals
Security

Quality

Content

61%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-referenced skill body whose main weaknesses are recurring redundancy (checklists duplicating headings, duplicated vLLM guidance) and absent validation/feedback loops in its batch workflows, which the rubric caps at 3. Progressive disclosure and reference structure are solid with only minor nesting/length concerns.

Suggestions

Remove the four 'Copy this checklist' blocks that duplicate the step headings immediately below them, and consolidate the vLLM speed guidance that appears in both Workflow 4 and the 'Common issues' section.

Add validation checkpoints to the batch workflows: check lm_eval's exit status in the model-comparison loop and the os.system() training-integration calls so failed runs are caught and can be retried instead of silently proceeding.

Fix the bash line-continuation blocks that place inline comments after the backslash (e.g., '--num_fewshot 5 \ # 5-shot evaluation (standard)') by moving comments above the command, so the blocks are truly copy-paste executable.

DimensionReasoningScore

Conciseness

The body is mostly dense, executable commands, but four 'Copy this checklist' blocks merely restate the step headings directly beneath them, vLLM speed guidance is duplicated between Workflow 4 and 'Common issues', and the standard tasks string appears repeatedly. This recurring redundancy exceeds anchor 4's 'minor instances that could be trimmed' without reaching anchor 2's pervasive padding, landing on anchor 3.

3 / 5

Actionability

Nearly everything is copy-paste ready — complete lm_eval invocations, full eval_checkpoint.sh and eval_all_models.sh scripts, expected JSON output, and a rendered comparison table covering HF, vLLM, quantized, and custom-checkpoint cases. However, bash line continuations broken by trailing inline comments ('--num_fewshot 5 \ # 5-shot evaluation (standard)'), the '...' placeholder in the Lightning callback, and the unimported glob usage are real executable gaps, matching anchor 4 rather than anchor 5.

4 / 5

Workflow Clarity

All four workflows have well-defined, per-step sequences with concrete commands, but no validation checkpoints anywhere: the batch model-comparison loop never checks lm_eval exit status, the training-integration examples call os.system() with unchecked return codes, and there is no verify-results step. The rubric caps batch operations without feedback loops at anchor 3, which applies here; the clear sequences keep it above anchor 2.

3 / 5

Progressive Disclosure

Advanced topics are cleanly delegated to five verified, one-level-deep, descriptively labeled references (benchmark-guide.md, custom-tasks.md, api-evaluation.md, distributed-eval.md, nemo-evaluator/overview.md), and the merged-tools table explains when to read each. It falls short of anchor 5 because the NeMo subtree nests a second reference level (overview.md -> its own references/) and the ~490-line body inlines detail (plotting code, table generation) that could live in references — anchor 4's 'minor organization gaps'.

4 / 5

Total

14

/

20

Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it answers both what and when explicitly, names concrete benchmarks and backends, and stakes out a distinct niche. The only gaps are a single-verb action framing and a few missing natural trigger variations.

DimensionReasoningScore

Specificity

Enumerates five named benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag), three backends (HuggingFace, vLLM, APIs), and a companion tool, but the action verbs are limited ('Evaluates', 'Supports', 'covers') rather than multiple distinct concrete actions. This sits at anchor 4 (several specifics, minor gaps), above anchor 3's 1-2 concrete actions and below anchor 5's comprehensive multi-action coverage.

4 / 5

Completeness

Explicitly answers 'what' ('Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag)... Supports HuggingFace, vLLM, APIs') and 'when' ('Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress') with concrete trigger phrases. The 'when' clause is explicit and specific, matching anchor 5 rather than anchor 4's merely-adequate trigger guidance.

5 / 5

Trigger Term Quality

'Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress' supplies natural phrases users would say, plus benchmark names users cite directly. Common variations like 'run benchmarks', 'model evaluation', or 'evaluate models' are missing, so it matches anchor 4 rather than anchor 5's comprehensive synonym coverage, and is clearly above anchor 3's partial keyword coverage.

4 / 5

Distinctiveness Conflict Risk

A clear niche (this specific harness, 'Industry standard used by EleutherAI, HuggingFace, and major labs') with distinct, unambiguous triggers; the NeMo Evaluator coverage is explicitly declared rather than vaguely overlapping. Minimal conflict risk matches anchor 5; anchor 4's 'minor overlap risk' would require a vaguer trigger set.

5 / 5

Total

18

/

20

Passed

Validation

81%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 13 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

relative_links

Relative link issues: 1 deeper-than-1-level

Warning

referenced_paths_exist

Referenced path issues: 2 deeper-than-1-level

Warning

Total

13

/

16

Passed

Repository

OpenLAIR/dr-claw

Path

skills/evaluation/lm-evaluation-harness/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.