CtrlK
BlogDocsLog inGet started
Tessl Logo

nemo-evaluator-sdk

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

68

Quality

82%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Critical

Do not install without reviewing

SKILL.md
Quality
Evals
Security

Quality

Content

65%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Highly actionable body with executable configs, CLI commands, and a Python API example, plus well-organized workflow sections and real advanced-topic references. Weaknesses are verbosity from repeated config/checklist blocks, missing verification checkpoints in batch-evaluation workflows, and a heavy inline body that dilutes the progressive-disclosure overview.

Suggestions

Deduplicate the config.yaml blocks: show the full `defaults/execution/target` structure once in Quick Start, then reference it (or show only the differing keys) in the per-workflow steps to reduce repetition and improve conciseness.

Add explicit validation/verification checkpoints to the evaluation workflows, e.g. after launch run `nemo-evaluator-launcher status <id>` and confirm a succeeded state before exporting results, so batch operations meet the workflow-clarity feedback-loop bar.

Move the CLI Reference table, Python API Usage, and the full Supported Harnesses table into a reference file (e.g. references/configuration.md or a new cli-reference.md) and link from a brief overview, slimming the main SKILL.md into a leaner overview for better progressive disclosure.

DimensionReasoningScore

Conciseness

Mostly efficient and free of concept-explanation padding, but it is tightened-poor: the same `defaults`/`execution`/`target` config block is reproduced in Quick Start, Workflow 1 Step 1, and again with minor variations, and each workflow repeats a checklist of generic "Step 1..Step 4" lines that add length without new information. This matches the level-2 anchor of mostly efficient but could be tightened, not the every-token-earns-its-place level 3.

2 / 3

Actionability

Provides fully executable, copy-paste-ready material throughout: complete `config.yaml` blocks, real `nemo-evaluator-launcher run/status/export` CLI commands with override flags, and a complete Python API example importing named dataclasses. It reaches the level-3 anchor of fully executable code/commands rather than the pseudocode/gaps of level 2.

3 / 3

Workflow Clarity

Workflows are clearly sequenced with per-workflow checklists (e.g. "Slurm Evaluation: Step 1..Step 4"), but the steps lack explicit validation/verification checkpoints — there is no "confirm status succeeded before exporting" or results-verification gate. Per the rubric guideline, batch operations (large-scale/HPC evaluation) missing verification steps cap workflow_clarity at 2.

2 / 3

Progressive Disclosure

The Advanced Topics section points to four real, one-level-deep references (execution-backends.md, configuration.md, adapter-system.md, custom-benchmarks.md) with clear markdown links, but the ~480-line body itself carries a full CLI reference table, Python API, full harness table, and several complete config reproductions inline rather than acting as a lean overview. This fits the level-2 anchor where some content that should be separate is inline, rather than the appropriately-split level 3.

2 / 3

Total

9

/

12

Passed

Description

100%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that states concrete capabilities, names representative benchmarks and platforms, and includes an explicit 'Use when' trigger in third-person voice. It clearly answers both what the skill does and when to invoke it with low conflict risk.

DimensionReasoningScore

Specificity

Names multiple concrete actions and specific artifacts: "Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution" and "reproducible benchmarking", matching the level-3 anchor of listing several specific concrete actions.

3 / 3

Completeness

It explicitly answers what ("Evaluates LLMs across 100+ benchmarks...") and when ("Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms"), satisfying the explicit-trigger requirement that otherwise caps completeness at 2.

3 / 3

Trigger Term Quality

The clause "Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms" plus named benchmarks (MMLU, HumanEval, GSM8K) gives good coverage of natural terms a user would say when seeking LLM evaluation; it is not just generic jargon, so it clears level 3 rather than the partial-coverage level 2.

3 / 3

Distinctiveness Conflict Risk

The multi-backend HPC/Docker/cloud enterprise-evaluation niche with named harnesses is a clear, distinct trigger set unlikely to fire for unrelated skills; it uses third-person voice ("Evaluates") with no first/second-person penalty, so it is not the merely-somewhat-specific level 2.

3 / 3

Total

12

/

12

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository
Orchestra-Research/AI-Research-SKILLs
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.