nemo-evaluator-sdk

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

Quality

82%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Critical

Do not install without reviewing

Quality

Content

65%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable with copy-paste configs, commands, and a Python example, plus well-signaled one-level references. Weaker on conciseness and progressive disclosure due to length, duplicated config blocks, and reference-grade material kept inline, and on workflow clarity because batch evaluation workflows lack explicit validation checkpoints.

Suggestions

De-duplicate the config block shown in both Quick Start and Workflow 1 Step 1, and move the CLI reference table, configuration override examples, and Python API into reference files to tighten the body and improve progressive disclosure.

Add an explicit validation/verification step to each evaluation workflow (e.g., confirm results.yml exists and task counts match expectations before declaring success), with a fix-and-reretry loop on failure.

Keep only a minimal getting-started config in SKILL.md and point to references/configuration.md for full options, reducing the inline YAML shown across multiple workflows.

Dimension	Reasoning	Score
Conciseness	Mostly concrete and accurate, but ~485 lines with duplicated config blocks (Quick Start repeats nearly verbatim in Workflow 1 Step 1) and inline reference-grade material could be tightened, fitting level-2 rather than the lean level-3.	2 / 3
Actionability	Provides fully executable, copy-paste-ready guidance — complete YAML configs, concrete bash commands, and a complete Python API example — matching the level-3 anchor rather than the incomplete level-2.	3 / 3
Workflow Clarity	Workflows have checklists and sequenced steps, but these are batch evaluation runs with no explicit validation/feedback checkpoints within the flow (only a separate troubleshooting section), so per the rubric note workflow_clarity is capped at 2.	2 / 3
Progressive Disclosure	Four real, one-level-deep, clearly signaled references exist under 'Advanced Topics', but substantial reference-worthy content (CLI reference table, configuration override examples, Python API, duplicated full configs) remains inline, so content is not fully 'appropriately split' as the level-3 anchor requires.	2 / 3
	Total	9 / 12 Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that states concrete capabilities, names specific benchmarks and backends, and includes an explicit 'Use when' trigger. The only mild weakness is the slightly marketing-flavored tagline 'enterprise-grade platform with container-first architecture', but it does not undermine specificity or completeness.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete capabilities with named benchmarks — "Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution" — matching the level-3 anchor rather than the partial level-2.	3 / 3
Completeness	Explicitly answers both what ("Evaluates LLMs across 100+ benchmarks...") and when ("Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms"), satisfying the level-3 anchor with an explicit trigger clause.	3 / 3
Trigger Term Quality	Covers natural terms a user would say — benchmark names (MMLU, HumanEval, GSM8K) plus "scalable evaluation", "local Docker", "Slurm HPC", "cloud platforms" — giving good coverage rather than missing common variations.	3 / 3
Distinctiveness Conflict Risk	Occupies a clear niche — NVIDIA NeMo enterprise benchmarking on Slurm/Docker/cloud with named harnesses — with distinct triggers unlikely to fire for unrelated skills, rather than the overlapping level-2 case.	3 / 3
	Total	12 / 12 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/nemo-evaluator/SKILL.md
Commit: 52845c3

Reviewed: about 13 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.