CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

evaluation-nemo-evaluator

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

Quality

80%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./cli-tool/components/skills/ai-research/evaluation-nemo-evaluator/SKILL.md
SKILL.md
Quality
Evals
Security

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable content with copy-paste-ready configs, commands, and troubleshooting, and a well-organized one-level-deep reference bundle. Weaknesses are moderate redundancy (repeated config block, checklists restating headings) and main-body bulk that could be thinned into the existing references.

Suggestions

Drop the config block duplicated between Quick Start and Workflow 1 Step 1 (keep one and cross-reference it), and replace per-workflow checklists that restate step headings with a single one-line goal statement.

Move the CLI Reference table, Configuration Override Examples, and Python API sections into a reference file (e.g. references/configuration.md or a new references/cli-api.md) to slim the main body toward an overview.

Add an explicit results-validation step before export/comparison (e.g., confirm results.yml exists and metrics parsed for every task, retry failed tasks) to close the workflow feedback loop.

DimensionReasoningScore

Conciseness

Mostly efficient — commands and config dominate with little conceptual padding — but the Quick Start config block is repeated nearly verbatim in Workflow 1 Step 1, and each workflow's checklist box restates its own step headings, adding ~40 lines of duplication. Not a 4 because the repeated config block is more than a minor trim; well above anchor 2 since there is no conceptual over-explanation.

3 / 5

Actionability

Fully executable throughout: complete runnable config.yaml examples, exact CLI invocations with override flags (`-o +evaluation.nemo_evaluator_config.config.params.limit_samples=10`), exact result paths (`results/<invocation_id>/<task>/artifacts/results.yml`), a Python API example with imports, and concrete troubleshooting commands. Covers the common cases (run, monitor, export, compare) copy-paste ready.

5 / 5

Workflow Clarity

Each of the four workflows has a clear numbered sequence, an upfront checklist, and status/verification steps (Step 4 'Check results', 'Monitor job status' with status/info/kill), plus a Common Issues section for error recovery. Not a 5 because there is no explicit validation of outputs or feedback loop (e.g., verify results.yml is complete/sane before exporting or comparing); the batch evaluation workflows do include status checks, so the missing-validation cap does not apply.

4 / 5

Progressive Disclosure

Good structure: Advanced Topics points to four clearly labeled, verified one-level-deep reference files (execution-backends.md, configuration.md, adapter-system.md, custom-benchmarks.md), none of which nest further references. Not a 5 because the ~480-line body inlines substantial reference-grade material (CLI Reference table, Configuration Override Examples catalog, Python API section, full harness table) that an overview-first SKILL.md would split out.

4 / 5

Total

16

/

20

Passed

Description

78%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description with a concrete 'what', an explicit 'Use when' clause, distinctive niche terminology, and natural benchmark-name triggers. Main weakness is the 'when' clause framing usage around scalable/HPC infrastructure, which under-covers simpler single-benchmark evaluation requests.

Suggestions

Broaden the 'Use when' clause to cover routine benchmarking requests, e.g. 'Use when running LLM benchmarks/evals (MMLU, GSM8K, HumanEval) on any OpenAI-compatible endpoint, or at scale on Docker, Slurm HPC, or cloud.'

Add common user synonyms such as 'evals', 'model evaluation', or 'benchmark harness' to the trigger vocabulary.

Trim the marketing phrase 'NVIDIA's enterprise-grade platform with container-first architecture' to the concrete differentiator ('containerized, reproducible evaluation') to reduce buzzword load.

DimensionReasoningScore

Specificity

Names the concrete action ('Evaluates LLMs across 100+ benchmarks from 18+ harnesses') with specifics (MMLU, HumanEval, GSM8K, safety, VLM) and concrete execution backends (local Docker, Slurm HPC, cloud). Not a 5 because it centers on a single action verb rather than the multiple distinct concrete actions of the top anchor; clearly above anchor 3 since coverage of scope and backends is detailed.

4 / 5

Completeness

Explicitly answers both: what ('Evaluates LLMs across 100+ benchmarks... with multi-backend execution') and when ('Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms'). Not a 5 because the 'when' clause is gated on scalability/infrastructure scenarios and would not clearly capture routine requests like 'run MMLU on my model'; well above anchor 3 since a real 'Use when' trigger is present.

4 / 5

Trigger Term Quality

Includes strong natural terms users would say — 'benchmarks', 'MMLU', 'HumanEval', 'GSM8K', 'Slurm HPC', 'reproducible benchmarking'. Not a 5 because common variations like 'evals', 'model evaluation', 'test model performance', or 'benchmark harness' are missing; well above anchor 3's partial keyword coverage.

4 / 5

Distinctiveness Conflict Risk

Clear niche (NVIDIA NeMo enterprise LLM benchmarking) with distinct triggers (NeMo, Slurm HPC, MMLU/HumanEval/GSM8K, NGC containers) — minimal overlap risk with generic evaluation or document skills. Fits the anchor-5 example's pattern of a distinct tool-specific trigger set.

5 / 5

Total

17

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

davila7/claude-code-templates

Path

cli-tool/components/skills/ai-research/evaluation-nemo-evaluator/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.