evaluation

Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on "evaluate model", "benchmark accuracy", "run MMLU", "evaluate quantized model", "run nel". Handles deployment, config generation, and evaluation execution. Not for quantizing models (use ptq), deploying/serving models (use deployment), or comparing completed baseline-vs-quantized results (use compare-results).

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

77%Weight 40%Scale 1-3

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-sequenced operational runbook with strong validation gates and rich domain-specific gotchas. Its weaknesses are token efficiency (length, repetition, rationale-padding) and progressive disclosure (a monolithic body that references several recipe/common paths absent from the bundle).

Suggestions

Move the nel-next path and the vLLM cross-check / env-var-prefix rules into reference files to slim the body, and de-duplicate the .env-location rationale (currently restated in Step 1 and Step 8) into a single statement.

Reconcile referenced paths with the bundle: either ship the recipes/ (example_eval.yaml, recipes/tasks/aa/*, env.example) and skills/common/* files the body depends on, or replace those references with inline minimal snippets so navigation is not broken.

Tighten the .env secret-safety and workspace-root rationale into imperative rules ('create .env at workspace root via cp; never Read/Write/Edit it') to cut defensive justification while preserving the constraint.

Dimension	Reasoning	Score
Conciseness	The ~400-line body is dense and mostly high-signal domain gotchas Claude would not know, but it is not lean: the .env-location rationale (Step 1) is padded with justification, and the nel-next path, MLflow literal rule, and .env setup recur across multiple steps — matching the 'mostly efficient but could be tightened' anchor rather than 'every token earns its place'.	2 / 3
Actionability	Fully executable commands throughout — `nel run --config <path> --dry-run`, `nel skills build-config --execution ...`, `ssh <host> "grep ..."`, `sbatch run.sub` — plus concrete YAML snippets, exact flags (e.g. `--enable-expert-parallel`), exact image tags (`vllm/vllm-openai:v0.19.1-cu130`), and exact env-var prefixes (`host:`/`lit:`/`runtime:`) that are copy-paste ready.	3 / 3
Workflow Clarity	A clearly sequenced Step 0–9 workflow with an up-front checklist, explicit gated validation (dry-run → canary → full in Step 8), and a feedback loop ('If the canary fails, fix and rerun the canary — don't skip to full'), plus error-recovery guidance for preemption and walltime chaining.	3 / 3
Progressive Disclosure	The seven references/.md files are well-signaled and one level deep, but the body also leans heavily on recipes/ (example_eval.yaml, recipes/tasks/aa/, env.example) and skills/common/* paths that do not exist in the bundle, breaking navigation, and much detail that could be split (nel-next path, vLLM cross-check rules) is inline — matching the 'some structure, references present but not cleanly resolved, content that should be separate is inline' anchor.	2 / 3
	Total	10 / 12 Passed

Description

100%Weight 40%Scale 1-3

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is exemplary: third-person voice, concrete actions, explicit natural-language triggers covering both 'what' and 'when', and strong negative delimitation against sibling skills. It satisfies every judging guideline with no fluff.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions on a named tool — 'Handles deployment, config generation, and evaluation execution' alongside 'Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL)' — matching the multiple-specific-actions anchor rather than the partial 'Names domain and some actions' anchor.	3 / 3
Completeness	Clearly answers both what ('Evaluates accuracy ... using NEL ... Handles deployment, config generation, and evaluation execution') and when ('Triggers on ...'), with explicit trigger guidance present — not the 'when missing or only implied' level 2 case.	3 / 3
Trigger Term Quality	Explicit natural triggers a user would say — 'evaluate model', 'benchmark accuracy', 'run MMLU', 'evaluate quantized model', 'run nel' — give good coverage of natural phrasings, beyond the single-keyword anchor.	3 / 3
Distinctiveness Conflict Risk	The 'Not for quantizing models (use ptq), deploying/serving models (use deployment), or comparing ... (use compare-results)' clause carves a clear niche and explicitly disambiguates from sibling skills, unlikely to trigger for the wrong skill.	3 / 3
	Total	12 / 12 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: NVIDIA/Model-Optimizer
Path: .agents/skills/evaluation/SKILL.md
Commit: 87c9f8c

Reviewed: 15 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.