nemo-evaluator-sdk

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

Quality

84%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Critical

Do not install without reviewing

Quality

Content

68%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable and well-structured with genuine reference files for deep topics, but it carries redundant config/checklist repetition and lacks explicit validation feedback loops for its batch evaluation workflows. Tightening the duplication and adding verify-before-proceed checkpoints would lift the weaker dimensions.

Suggestions

Remove the duplicated config block in Workflow 1 Step 1 (it is identical to the Quick Start example) and reference back to it instead, to improve conciseness.

Add an explicit validation checkpoint in each workflow after running an evaluation — e.g. check `status` succeeded and confirm `results.yml` exists before exporting or comparing — to satisfy the batch-operation feedback-loop expectation.

Move the CLI Reference and/or Python API Usage sections into a reference file linked from the body, leaving the SKILL.md as a tighter overview.

Dimension	Reasoning	Score
Conciseness	The body is mostly efficient with executable blocks, but the Quick Start config is repeated verbatim in Workflow 1 Step 1, and the per-workflow checklists restate the step headings that immediately follow, adding avoidable tokens.	3 / 5
Actionability	Copy-paste ready configs, CLI commands, override examples, and a Python API snippet cover the common cases concretely with specific model IDs and URLs.	5 / 5
Workflow Clarity	Workflows have clear step sequences with checklists, but evaluation runs are batch operations and the steps lack explicit validation/feedback loops (validate -> fix -> retry); per the batch-operation cap, this caps the score at 3.	3 / 5
Progressive Disclosure	Advanced topics are cleanly offloaded to four real one-level-deep reference files (execution-backends.md, configuration.md, adapter-system.md, custom-benchmarks.md) with clear links, but the body itself is long and retains inline CLI reference, override examples, and Python API content that could be split out.	4 / 5
	Total	15 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is third-person, concrete, and explicitly pairs a 'what' statement with a 'Use when...' trigger clause, covering specific benchmarks and execution backends. It is concise yet comprehensive with minimal conflict risk.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete capabilities ('Evaluates LLMs across 100+ benchmarks from 18+ harnesses', 'multi-backend execution', 'reproducible benchmarking') with specific benchmark names (MMLU, HumanEval, GSM8K, safety, VLM), matching the comprehensive-coverage anchor.	5 / 5
Completeness	Explicitly answers both 'what' ('Evaluates LLMs across 100+ benchmarks...') and 'when' ('Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms'), matching the concrete-trigger anchor.	5 / 5
Trigger Term Quality	Natural user phrases and synonyms are well covered ('evaluate', 'benchmarking', 'benchmarks', 'MMLU', 'HumanEval', 'GSM8K', 'Slurm HPC', 'Docker', 'cloud'), aligning with the comprehensive-synonym anchor.	5 / 5
Distinctiveness Conflict Risk	A clear NVIDIA NeMo enterprise benchmarking niche with multi-backend triggers; the body's 'When to Use vs Alternatives' section further reduces overlap risk.	5 / 5
	Total	20 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: OpenLAIR/dr-claw
Path: skills/evaluation/nemo-evaluator/SKILL.md
Commit: 4b35756

Reviewed: 5 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.