CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

nemo-evaluator-sdk

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

61

Quality

77%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./backend/cli/skills/ml-training/nemo-evaluator/SKILL.md
SKILL.md
Quality
Evals
Security

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill body with copy-paste configs, commands, and a complete CLI reference, plus properly signaled one-level references that all exist on disk. Its weaknesses are token inefficiency from a duplicated Quick Start config and redundant checklists, and the absence of explicit validation checkpoints in batch evaluation workflows.

Suggestions

Remove the near-verbatim duplication of the Quick Start config in Workflow 1 Step 1 (reference the Quick Start config and show only the delta), and drop or compress the checkbox checklists that restate step headings.

Add an explicit validation checkpoint before downstream steps, e.g., verify status output shows success and results.yml exists before running 'export <invocation_id>'.

Move the Python API usage, CLI reference table, and full harness table into a reference file to shorten SKILL.md toward a true overview.

DimensionReasoningScore

Conciseness

The Quick Start YAML config is repeated nearly verbatim in Workflow 1 Step 1, and the checkbox checklists ('- [ ] Step 1: Configure API endpoint') restate the headings that immediately follow without adding information. There is no conceptual padding of things Claude already knows, so it is above the verbose anchors, but the duplication and token-cost checklists are exactly the 'could be tightened' case.

3 / 5

Actionability

Every section is copy-paste executable: 'pip install nemo-evaluator-launcher', complete YAML configs, concrete CLI invocations ('nemo-evaluator-launcher run --config-dir . --config-name config') with real override flags, a CLI command table, and a working Python API snippet. The workflows cover the common cases (local, Slurm, model comparison, safety/VLM), matching the fully-executable anchor.

5 / 5

Workflow Clarity

Four workflows each present a clear numbered step sequence with concrete per-step commands, and Workflow 1 ends with 'Step 4: Check results' (status <invocation_id>) plus a Common Issues section for recovery. However, batch evaluation runs lack an explicit validate-before-proceeding loop (e.g., confirm results exist before export), so it does not reach the explicit-validation/feedback-loop anchor.

4 / 5

Progressive Disclosure

Good structure: an overview body with a clearly signaled 'Advanced Topics' section linking each of the four existing reference files (execution-backends.md, configuration.md, adapter-system.md, custom-benchmarks.md), one level deep. Minor gaps: the inline Python API section, CLI reference table, and harness table are material that could itself be moved to references, keeping SKILL.md at ~480 lines, which fits the good-structure-with-minor-gaps anchor rather than the cleanly split top anchor.

4 / 5

Total

16

/

20

Passed

Description

73%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description that answers both what and when with a clear NVIDIA-evaluation niche and named benchmarks. Its main weaknesses are a single-verb action set (evaluate, qualified by scope) and a when-clause oriented to infrastructure rather than natural user-intent triggers, which leaves specificity and completeness just below the top anchors.

Suggestions

Broaden the action verbs beyond 'evaluates' to concrete operations users might want, e.g., 'Runs benchmark suites, compares models, and exports results to MLflow/W&B'.

Rewrite the when-clause around user intent rather than infrastructure only, e.g., 'Use when the user wants to benchmark an LLM, compare models on MMLU/GSM8K/HumanEval, or run evals at scale on Docker, Slurm, or cloud'.

Add natural synonyms such as 'evals', 'harness', and 'model scoring' to improve trigger-term coverage.

DimensionReasoningScore

Specificity

Names the domain concretely ('Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution') but the action set reduces to a single verb qualified by scope; it does not enumerate several distinct concrete operations, so it sits at the 1-2-concrete-actions anchor rather than the several-specific-actions anchor.

3 / 5

Completeness

Both parts are explicitly present: what ('Evaluates LLMs across 100+ benchmarks... with multi-backend execution') and when ('Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms'). The when-clause covers infrastructure contexts rather than user-intent triggers like 'run MMLU' or 'benchmark a model', so it matches the 'when could be more specific' anchor, not the concrete-trigger-phrases anchor.

4 / 5

Trigger Term Quality

Good natural keyword coverage ('benchmarks', 'MMLU, HumanEval, GSM8K', 'local Docker, Slurm HPC', 'cloud platforms') that users would plausibly say, but common variations like 'eval', 'harness', or 'score my model' are missing, fitting the good-coverage-with-a-few-gaps anchor rather than the comprehensive-synonyms anchor.

4 / 5

Distinctiveness Conflict Risk

The description carves a clear niche ('NVIDIA's enterprise-grade platform', '18+ harnesses', 'Slurm HPC') with named-benchmark triggers and minimal overlap risk against generic evaluation skills, matching the clear-niche anchor.

5 / 5

Total

16

/

20

Passed

Validation

81%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 13 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

skill_md_line_count

SKILL.md is long (504 lines); consider splitting into references/ and linking

Warning

metadata_version

'metadata.version' is missing

Warning

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

13

/

16

Passed

Repository

synthetic-sciences/openscience

Path

backend/cli/skills/ml-training/nemo-evaluator/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.