evaluating-code-models

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

Quality

82%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

65%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is highly actionable with executable commands and clear workflow sequences, but it is padded with reference-table and troubleshooting content that duplicates existing bundle files which are never linked, and its code-execution workflows lack explicit validation checkpoints. Tightening the inline material and pointing to the existing references would lift both conciseness and progressive disclosure.

Suggestions

Link the existing bundle files — replace the inline 'Supported Benchmarks' table and 'Common Issues' section with pointers to references/benchmarks.md and references/issues.md so the body stays a lean overview.

Add validation checkpoints to the evaluation workflows — e.g., after '--save_generations', verify the generations file exists and is non-empty before running Docker evaluation; after evaluation, confirm every requested task appears in the results JSON.

Trim or move the Command Reference and Hardware Requirements tables to a reference file; they add tokens without adding decision-making guidance Claude does not already have access to.

Dimension	Reasoning	Score
Conciseness	The body is mostly efficient with concrete commands, but it inlines substantial content that Claude largely already knows or could be split out — full benchmark descriptions, a hardware-requirements table, and a 20-row command reference — producing noticeable verbosity relative to a leaner overview.	3 / 5
Actionability	Commands are copy-paste ready and complete — 'accelerate launch main.py --model bigcode/starcoder2-7b --tasks humaneval ... --save_generations' plus a results JSON example and a runnable comparison script cover the common cases fully.	5 / 5
Workflow Clarity	Each workflow uses an explicit step-numbered checklist and sequenced steps, but operations that execute model-generated code are inherently risky/batch and the workflows lack explicit validation checkpoints (e.g., verifying generations were saved before the Docker eval, checking results JSON for missing tasks); per the batch-operation cap, this caps the score at 3.	3 / 5
Progressive Disclosure	Bundle files benchmarks.md, custom-tasks.md, and issues.md exist but are never referenced or linked from the body — the body inlines benchmark details and troubleshooting that overlap with those files, so structure exists but references are not signaled and content that belongs in separate files is duplicated inline.	3 / 5
	Total	14 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

This is a strong description that explicitly states concrete capabilities, named benchmarks, the pass@k metric, and a clear 'Use when' trigger clause with multiple natural phrases. It uses third-person voice, avoids vagueness, and is clearly distinct from other skills.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions and artifacts — 'Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics' names specific benchmarks, the metric, and the evaluation action, giving comprehensive coverage of what the skill does.	5 / 5
Completeness	Explicitly answers both 'what' (evaluates code generation models across named benchmarks with pass@k metrics) and 'when' ('Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality') with concrete trigger phrases.	5 / 5
Trigger Term Quality	The 'Use when' clause covers natural phrases users would say — 'benchmarking code models', 'comparing coding abilities', 'testing multi-language support', 'measuring code generation quality' — with strong synonym coverage and named benchmarks as natural keywords.	5 / 5
Distinctiveness Conflict Risk	The niche is distinct and specific — code generation model benchmarking via the BigCode Evaluation Harness with named benchmarks and the BigCode/HuggingFace leaderboard provenance — making conflict with other skills minimal.	5 / 5
	Total	20 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: foryourhealth111-pixel/Vibe-Skills
Path: bundled/skills/evaluating-code-models/SKILL.md
Commit: d5ae560

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.