evaluating-code-models

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

Quality

76%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./backend/cli/skills/ml-training/bigcode-evaluation-harness/SKILL.md

Quality

Content

70%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-structured skill with executable commands and clear stepwise workflows, but it is somewhat verbose due to repeated command blocks and inlines reference-grade material (issues, benchmarks) that should be offloaded to the existing reference files and linked.

Suggestions

Link the existing references from the body (e.g., 'See references/benchmarks.md' under Supported Benchmarks, 'See references/issues.md' under Common Issues) and move the bulk of the issues and benchmark-detail tables into those files to reduce inline duplication.

Reduce repeated near-identical `accelerate launch` blocks by showing one canonical example and noting only the flag differences for variants (quantized, custom model, multi-language).

Add an explicit validation/verification checkpoint in the multi-model comparison workflow (e.g., confirm each results JSON exists and contains expected pass@k keys before building the table).

Dimension	Reasoning	Score
Conciseness	Mostly efficient with executable commands and tables, but it repeats near-identical accelerate launch invocations across workflows and includes explanatory phrasing Claude already knows (e.g., restating benchmark counts already shown in the table).	3 / 5
Actionability	Provides fully executable, copy-paste-ready commands and scripts throughout (install, evaluate, Docker, model-comparison bash loop, pandas table generator), covering the common cases with concrete flag values.	5 / 5
Workflow Clarity	Each workflow uses explicit Step 1-4 checklists with concrete commands and a results-interpretation step, but validation/verification checkpoints are largely implicit; for batch model-comparison it relies on output inspection rather than an explicit verify step.	4 / 5
Progressive Disclosure	Has good section structure and three reference files exist, but the SKILL.md body inlines large amounts of content (benchmark table, command reference, hardware table, full issues section) that overlaps with references without signaling them, and references are not linked from the body.	3 / 5
	Total	15 / 20 Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person description with a clear 'Use when...' clause covering both what and when. It is specific and well-triggered, with only minor redundancy in the listed actions and slight overlap risk with general LLM benchmarking skills.

Dimension	Reasoning	Score
Specificity	Names the domain (code generation models) and several concrete actions (benchmarking, comparing coding abilities, testing multi-language support, measuring code generation quality), though the actions are somewhat overlapping rather than distinctly comprehensive.	4 / 5
Completeness	Clearly answers both 'what' (evaluates code generation models across named benchmarks with pass@k metrics) and 'when' ('Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality').	5 / 5
Trigger Term Quality	Includes natural user phrases like 'benchmarking code models', 'comparing coding abilities', and 'measuring code generation quality', with good keyword coverage (HumanEval, MBPP, pass@k), though it lacks file extensions and a few synonyms.	4 / 5
Distinctiveness Conflict Risk	The BigCode/code-model benchmarking niche is distinct and named with specific triggers, with only minor overlap risk against general lm-evaluation-harness usage.	4 / 5
	Total	17 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: synthetic-sciences/openscience
Path: backend/cli/skills/ml-training/bigcode-evaluation-harness/SKILL.md
Commit: fcd2e11

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.