nemo-evaluator-sdk

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Critical

Do not install without reviewing

Quality

Content

82%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill body built around executable configs and commands with clearly signaled one-level-deep references. Its main weaknesses are mild verbosity from redundant checklists and several reference-grade sections that would be better placed in bundle files.

Suggestions

Move the CLI Reference table, Python API Usage block, and Configuration Override Examples into a reference file (or extend configuration.md) and link to them from SKILL.md to tighten the overview and improve progressive disclosure.

Remove or compress the per-workflow checklists that restate the numbered steps immediately following them, since they duplicate content without adding guidance.

Add an explicit validation feedback loop to at least the batch evaluation workflows (e.g., after 'Run evaluation', check status and if failed, inspect logs then re-run with adjusted parallelism/limit_samples) to push workflow clarity toward anchor 5.

Dimension	Reasoning	Score
Conciseness	The body is dense with executable configs and commands rather than padded with concepts Claude already knows, but the per-workflow checklists restate the steps that immediately follow and a few sections (CLI Reference, Python API, override examples) could be trimmed or moved, so it sits at efficient-with-minor-trims rather than lean.	4 / 5
Actionability	Fully copy-paste-ready guidance throughout: complete config.yaml blocks, exact 'nemo-evaluator-launcher run --config-dir . --config-name config' commands, override flags, and a runnable Python API example covering the common standard/Slurm/comparison/safety cases.	5 / 5
Workflow Clarity	Four workflows are clearly sequenced with numbered steps and checklists, and each includes a verification step ('Check results', 'Monitor job status'), but explicit validate->fix->retry feedback loops are not spelled out inline, keeping it just below anchor 5.	4 / 5
Progressive Disclosure	Good structure with a clearly signaled, one-level-deep 'Advanced Topics' section linking to four verified reference files (execution-backends.md, configuration.md, adapter-system.md, custom-benchmarks.md); however, the CLI Reference table, Python API block, and Configuration Override Examples are reference-grade material inlined in SKILL.md rather than split out.	4 / 5
	Total	17 / 20 Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that clearly states capabilities and provides an explicit 'Use when' trigger with concrete platform contexts. It is highly specific and distinctive, with only minor room to broaden trigger-term synonyms.

Dimension	Reasoning	Score
Specificity	Lists several concrete specifics ('Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution') but the action set centers on a single verb (evaluate) rather than multiple distinct operations, so it stops short of the comprehensive multi-action anchor 5.	4 / 5
Completeness	Explicitly answers both 'what' (evaluates LLMs across 100+ benchmarks from 18+ harnesses with multi-backend execution) and 'when' via a concrete 'Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms' trigger clause, matching the anchor 5 example.	5 / 5
Trigger Term Quality	Strong keyword coverage including natural terms an ML practitioner would say ('benchmarks', 'MMLU', 'HumanEval', 'GSM8K', 'safety', 'Slurm HPC', 'Docker'), but misses common synonym phrasings like 'test my model' or 'score my model'.	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (NVIDIA enterprise-grade LLM benchmarking with container-first reproducible execution) with distinct triggers (MMLU/HumanEval/Slurm/Docker) and minimal overlap risk against generic eval skills.	5 / 5
	Total	18 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/nemo-evaluator/SKILL.md
Commit: 3a6c3a9

Reviewed: 3 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.