CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

evaluation-bigcode-evaluation-harness

Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.

Quality

74%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./cli-tool/components/skills/ai-research/evaluation-bigcode-evaluation-harness/SKILL.md
SKILL.md
Quality
Evals
Security

Quality

Content

53%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body delivers highly actionable, copy-paste-ready commands with well-sequenced workflows, but it functions as a monolith: it ignores its own reference bundle, inlines duplicated benchmark and troubleshooting content, and lacks validation checkpoints for long batch evaluations. The single broken Docker image reference is the main executability flaw.

Suggestions

Add explicit links to the existing bundle files — e.g., under Supported Benchmarks point to references/benchmarks.md, under a new Custom Tasks section point to references/custom-tasks.md, and under Common Issues point to references/issues.md — and trim the inlined duplicates (progressive_disclosure).

Insert verification checkpoints into the workflows: confirm the generations file exists and is non-empty before the Docker evaluation step, and confirm the metric output JSON was written with sane pass@k values before analyzing results (workflow_clarity).

Remove the duplicated benchmark descriptions (keep either the Workflow 1 Step 1 bullets or the Supported Benchmarks table, not both) and move the Hardware Requirements table to a reference file to cut inline tokens (conciseness).

Fix the MultiPL-E Docker command to use the full pulled image name (ghcr.io/bigcode-project/evaluation-harness-multiple) so the command is executable as written (actionability).

DimensionReasoningScore

Conciseness

The body is command-dense and avoids re-teaching known concepts, but benchmark descriptions are duplicated (Workflow 1 Step 1 bullet list vs. the Supported Benchmarks table) and the Common Issues section duplicates references/issues.md. Anchor 3 ('mostly efficient but could be tightened') fits; the duplication is more than the 'minor instances' of anchor 4.

3 / 5

Actionability

Nearly all guidance is copy-paste ready (install, standard/quantized/custom-model eval, Docker, a model-comparison bash script and pandas table). Not anchor 5's 'fully executable': the MultiPL-E workflow pulls 'ghcr.io/bigcode-project/evaluation-harness-multiple' but then runs 'docker run ... evaluation-harness-multiple', which fails without the full image name.

4 / 5

Workflow Clarity

Workflows have clear checklists and numbered steps, but none include verification checkpoints (e.g., confirm generations_multi.json exists before the Docker eval, confirm the results JSON was written) for batch GPU runs taking hours. Per the rubric's cap, batch operations without validation cannot score above 3 even though sequences are otherwise clear.

3 / 5

Progressive Disclosure

The bundle contains references/benchmarks.md, references/custom-tasks.md, and references/issues.md, but the body never references any of them — they are undiscoverable — while their content is inlined instead (Supported Benchmarks table, Common Issues). This matches anchor 2 ('content that clearly belongs in separate files is inlined'); it is below anchor 3 because references are not merely unclear, they are entirely absent from the body.

2 / 5

Total

12

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description with a concrete what-statement, an explicit and multi-trigger 'Use when' clause, and good natural keyword coverage. The only weaknesses are minor: slightly incomplete action coverage and small overlap risk with neighboring evaluation-harness skills.

DimensionReasoningScore

Specificity

Names concrete specifics — 'HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics' — plus action variants (benchmark, compare, test multi-language). Minor gaps in coverage (e.g., fill-in-the-middle evaluation from the body is absent), so anchor 4 fits better than the comprehensive anchor 5 or the thinner anchor 3.

4 / 5

Completeness

Explicitly answers both: what ('Evaluates code generation models across ... with pass@k metrics') and when ('Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality'). This matches the anchor-5 pattern of concrete what + explicit trigger phrases; the when-clause is already specific, so anchor 4's 'could be more explicit' doesn't apply.

5 / 5

Trigger Term Quality

'benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality' gives good synonym coverage alongside benchmark names users would naturally say. A few natural terms are missing (e.g., 'code LLM evaluation', 'pass@1'), keeping it below anchor 5.

4 / 5

Distinctiveness Conflict Risk

Clear niche (code-generation benchmarking via named benchmarks) that would not trigger for general LLM evals, but 'benchmarking code models' overlaps slightly with closely related skills like a generic lm-evaluation-harness or EvalPlus skill — anchor 4's 'minor overlap risk with closely related skills'.

4 / 5

Total

17

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

15

/

16

Passed

Repository

davila7/claude-code-templates

Path

cli-tool/components/skills/ai-research/evaluation-bigcode-evaluation-harness/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.