CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

evaluating-code-models

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

64

Quality

81%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

71%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable body with copy-paste-ready commands, clear sequenced workflows, and useful troubleshooting and decision guidance. Its main weaknesses are repetition of near-identical command blocks and a broken progressive-disclosure structure: three substantial reference files sit orphaned while their content (benchmarks detail, common issues) is duplicated inline.

Suggestions

Link the existing reference files from the body (e.g. 'For all 15+ benchmarks see [benchmarks.md](references/benchmarks.md); for troubleshooting see [issues.md](references/issues.md); for custom tasks see [custom-tasks.md](references/custom-tasks.md)') and trim the inlined 'Supported Benchmarks' table and 'Common Issues' section to summaries.

Reduce repetition of the `accelerate launch` command: show one canonical invocation once and note only the flags that vary per scenario (quantization, instruction tokens, generation-only).

Add validation checkpoints to the workflows, e.g. verify the metrics JSON exists and parse it before generating the comparison table, and `mkdir -p results` at the top of the multi-model eval script.

DimensionReasoningScore

Conciseness

The body is mostly dense, executable value (commands, tables, step lists) with little padding of concepts Claude already knows, but it could be tightened: the `accelerate launch` invocation is repeated ~6 times with overlapping flag sets across Quick Start, Workflow 1 Steps 2–3, and Workflow 3, and the 'Supported Benchmarks' table and 'Common Issues' section duplicate content that already lives in references/benchmarks.md and references/issues.md. This is the 'mostly efficient but could be tightened' anchor rather than the noticeably padded one below it.

3 / 5

Actionability

Fully executable, copy-paste-ready commands throughout: install sequence, complete `accelerate launch` invocations for standard/quantized/custom models, Docker evaluation for MultiPL-E, instruction-token configuration, a runnable bash comparison script, and a Python snippet to build a comparison table. Specific examples cover the common cases (single eval, multi-language, instruction models, multi-model comparison).

5 / 5

Workflow Clarity

Each workflow has a checklist and explicitly numbered, well-sequenced steps, and the 'Common Issues' section provides error-recovery guidance (e.g. OOM → quantize/reduce batch; hanging execution → generate on host, evaluate in Docker). It falls short of 5 because there are no explicit validation checkpoints — e.g. verify the metrics file exists before building the comparison table, or confirm `results/` exists before the loop script writes to it.

4 / 5

Progressive Disclosure

The body has good section structure (Quick Start, Workflows, When to Use vs Alternatives, Command Reference), but three reference files exist — references/benchmarks.md (393 lines), references/custom-tasks.md (424 lines), references/issues.md (394 lines) — and none is ever mentioned in the body. Meanwhile 'Supported Benchmarks' and 'Common Issues' content is inlined, duplicating those files. This matches the 3 anchor: some structure, references present but not clearly signaled, content that should be separate is inline.

3 / 5

Total

15

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that explicitly states what the skill does and when to use it, with concrete trigger phrases and specific benchmark/metric names users would naturally say. Its only weaknesses are a slightly repetitive action list and mild overlap risk with adjacent code-evaluation harness skills.

DimensionReasoningScore

Specificity

Names concrete specifics — 'Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics' — plus actions like 'benchmarking', 'comparing coding abilities', 'testing multi-language support', 'measuring code generation quality'. It lists several specific actions with named benchmarks and metric, though the action set is less comprehensive than the 5 anchor's enumeration of distinct capabilities.

4 / 5

Completeness

Explicitly answers both: what — 'Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics'; when — 'Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality'. This matches the 5 anchor: concrete 'what' plus a 'Use when...' clause with concrete trigger phrases.

5 / 5

Trigger Term Quality

Good natural keyword coverage: 'benchmarking code models', 'comparing coding abilities', 'testing multi-language support', plus proper nouns users actually say (HumanEval, MBPP, pass@k, BigCode). A few natural variants (e.g. 'code LLM evaluation', 'coding performance') are missing, keeping it below the comprehensive 5 anchor.

4 / 5

Distinctiveness Conflict Risk

Benchmark names (HumanEval, MBPP, MultiPL-E) and pass@k/BigCode create a clear niche, but the skill's domain has closely related alternatives (EvalPlus, SWE-bench, lm-evaluation-harness) where phrases like 'comparing coding abilities' or 'measuring code generation quality' could overlap. Mostly distinct with minor overlap risk — the 4 anchor.

4 / 5

Total

17

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

OpenLAIR/dr-claw

Path

skills/evaluation/bigcode-evaluation-harness/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.