CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

devtu-benchmark-harness

Continuous improvement system for ToolUniverse tools, skills, and plugin. Run benchmarks, diagnose failures, route fixes to devtu skills, retest. Use after skill optimization, tool additions, or as regression check.

68

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

77%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-sequenced harness skill whose commands and validation gates are excellent. Its main weakness is bundle hygiene: an existing reference file is orphaned, detail that belongs there is inlined in SKILL.md, and script inventory is only partially documented.

Suggestions

Link references/benchmark-guide.md from the body (e.g., in the 'Available benchmarks' section) so the bundled benchmark/category detail is discoverable instead of inlined.

Move the 'Known Failure Patterns' table and the 7-strategy grader detail into references/ (or the benchmark guide) and keep one-line pointers in SKILL.md to reduce body size and duplication with Step 2's category mapping.

Reconcile the runner path inconsistency ('skills/evals/run_benchmark.py' vs 'skills/devtu-benchmark-harness/scripts/run_eval.py') and document or remove the 9 unreferenced bundled scripts (analyze_failures.py, generate_report.py, skill_routing_test.py, solve_one.sh, etc.).

DimensionReasoningScore

Conciseness

The body is dense with commands and tables and assumes Claude's competence (no concept tutorials), but has minor trimmable material: the category-to-skill mapping repeats across Step 2 and the Fix-routing table, and the 7-strategy grader detail could be more compact. Not 5: not every token earns its place; not 3: there is no padded over-explanation.

4 / 5

Actionability

Nearly all guidance is copy-paste-ready (run_harness_loop.sh invocation, run_eval.py with flags, diagnose workflow, retest commands). Not 5: minor gaps — the APPEND_CONVENTIONS example calls 'skills/evals/run_benchmark.py' while the runner elsewhere is 'scripts/run_eval.py', and several bundled scripts are never mentioned.

4 / 5

Workflow Clarity

The 5-step loop (RUN, ANALYZE, DIAGNOSE, FIX, RETEST) is clearly sequenced with explicit validation checkpoints (check_memorization.py gate before accepting edits, GT-vs-agent root-cause verification, retest comparison) and a full error-recovery feedback loop. Not 4: checkpoints and recovery loops are explicit, satisfying the batch-operations validation requirement.

5 / 5

Progressive Disclosure

Sections are well-organized, but the bundled references/benchmark-guide.md is never linked from the body while overlapping detail (known failure patterns, category tables, grader strategies) is inlined instead of split out, and 9 of 14 scripts are undocumented. Not 4: a provided reference is completely unsignaled; not 2: structure itself is good and nothing is deeply nested.

3 / 5

Total

16

/

20

Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that pairs concrete loop actions with an explicit 'Use after...' trigger clause and is clearly scoped to the ToolUniverse/devtu ecosystem. The only weaknesses are an abstract first sentence and a few missing natural trigger synonyms.

DimensionReasoningScore

Specificity

"Run benchmarks, diagnose failures, route fixes to devtu skills, retest" lists several concrete actions in the ToolUniverse domain. Not 5: the abstract opener "Continuous improvement system" and missing steps (analyze, anti-memorization guard) leave minor coverage gaps.

4 / 5

Completeness

Explicitly answers both: what ("Run benchmarks, diagnose failures, route fixes to devtu skills, retest") and when ("Use after skill optimization, tool additions, or as regression check") with concrete trigger phrases. Not 4 because the when-clause is specific and fully explicit rather than vague or weakly implied.

5 / 5

Trigger Term Quality

Natural terms a user would say include "Run benchmarks", "regression check", "skill optimization", "tool additions", and "retest". Not 5: common variations like "eval", benchmark names, or "scores" are missing; not 3 because coverage is genuinely good, not merely 'some relevant keywords'.

4 / 5

Distinctiveness Conflict Risk

"ToolUniverse tools, skills, and plugin" plus "route fixes to devtu skills" carves a clear niche (benchmark-driven improvement) distinct from sibling fix/optimize skills, with benchmark/regression triggers that would not fire for the wrong skill. Not 4: overlap risk is minimal.

5 / 5

Total

18

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

referenced_paths_exist

Referenced path issues: 2 missing

Warning

Total

15

/

16

Passed

Repository

mims-harvard/ToolUniverse

Path

skills/devtu-benchmark-harness/SKILL.md

Revision

83f02cc

Reviewed

Table of Contents

DiscoveryImplementationValidation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.

Claim skill