CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

devtu-benchmark-harness

Continuous improvement system for ToolUniverse tools, skills, and plugin. Run benchmarks, diagnose failures, route fixes to devtu skills, retest. Use after skill optimization, tool additions, or as regression check.

Quality

83%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

77%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body delivers a clearly sequenced, validated feedback loop with strong executable guidance — routing tables, exact commands, and explicit retest/memorization checkpoints are exemplary. Its main weaknesses are progressive disclosure (the existing references/benchmark-guide.md is never referenced, and grader/failure-pattern detail is inlined) and minor executability gaps (commented line-continuations in the Step 1 command block).

Suggestions

Link references/benchmark-guide.md from the body (e.g., a '## Benchmark details' section: 'See [benchmark-guide.md](references/benchmark-guide.md) for per-benchmark subtask tables, category breakdowns, and score history') so the existing reference file is discoverable instead of orphaned.

Move the 7 grader strategies, unicode normalization notes, and Known Failure Patterns table into references/ (or into benchmark-guide.md) and keep only the invocation command plus a one-line summary in SKILL.md.

Fix the Step 1 run_eval.py code block so it is copy-paste executable — put the option comments (bixbench | lab-bench | custom, plugin-only | baseline-only | comparison) on their own lines above the command instead of after line-continuation backslashes, which currently break the shell syntax.

DimensionReasoningScore

Conciseness

The body is command-dense and mostly lean — every step carries an exact invocation, and explanations are limited to non-obvious project-specific rationale (e.g., the router-only architecture "1 auto-matchable skill... 114 skills × 500 chars = 57K exceeds budget"). A few sections could be trimmed or moved: the 7 grader strategies with unicode-normalization detail and the Known Failure Patterns table are reference material inlined into the body, fitting the 4 anchor ("Efficient; minor instances of over-explanation that could be trimmed") rather than the every-token-earns-its-place 5 anchor.

4 / 5

Actionability

Highly actionable: copy-paste bash/python commands with flags, a diagnosis→skill routing table with literal Skill() invocations, and a worked example including the exact message to pass to the fix skill. It stops short of 5 because some commands are not executable as written — the Step 1 run_eval.py block places comments after line-continuation backslashes ("--benchmark bixbench \ # bixbench | lab-bench | custom"), which breaks the shell syntax — and several referenced scripts (skills/evals/run_benchmark.py, scripts/build-plugin.sh) are external to the bundle, matching the 4 anchor ("concrete code or commands with minor gaps").

4 / 5

Workflow Clarity

The 5-step loop (RUN → ANALYZE → DIAGNOSE → FIX → RETEST) is explicitly sequenced with one section per step, a one-command orchestrated runner, a numbered fix workflow, and an explicit feedback loop (retest failures, count wrong→correct flips, repeat). Validation checkpoints are present for the batch operations: the anti-memorization guard before accepting any skill edit, root-cause verification ("Run the script yourself... it's a GT issue, not an agent error"), and retest verification — matching the 5 anchor ("explicit validation steps; feedback loops for error recovery").

5 / 5

Progressive Disclosure

The bundle contains references/benchmark-guide.md (per-benchmark subtask tables, category breakdowns, score history) and 13 scripts, but the body never links or mentions the guide — it is undiscoverable from SKILL.md, and its subject matter is partially duplicated inline (the Available benchmarks table, Known Failure Patterns, grader strategy detail) where it belongs in the reference file. Scripts are well-signaled via inline commands, but the orphaned reference and inline reference-material match the 3 anchor ("references present but not clearly signaled; content that should be separate is inline") better than the 4 anchor.

3 / 5

Total

16

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it states the domain, lists four concrete loop actions, and includes an explicit 'Use after...' clause with concrete trigger conditions. Trigger coverage is good, and the harness role is well distinguished from the fix-skills it routes to. The only room for improvement is slightly broader action/keyword coverage.

DimensionReasoningScore

Specificity

Lists several concrete actions — "Run benchmarks, diagnose failures, route fixes to devtu skills, retest" — anchored to a named domain ("ToolUniverse tools, skills, and plugin"). It falls just short of the 5 anchor because coverage has minor gaps (e.g., the analyze-results step and grader re-scoring from the body are absent), matching "Lists several specific actions; minor gaps in coverage" rather than comprehensive coverage.

4 / 5

Completeness

Explicitly answers both: what — "Run benchmarks, diagnose failures, route fixes to devtu skills, retest" — and when — "Use after skill optimization, tool additions, or as regression check", which is a concrete Use-when clause with three explicit trigger conditions. This matches the 5 anchor ("Clearly and explicitly answers both what AND when with concrete trigger phrases"); the 4 anchor applies only when the when-clause is vague or implicit, which is not the case here.

5 / 5

Trigger Term Quality

Contains natural trigger phrases users would say: "run benchmarks", "regression check", "failures", "skill optimization", "tool additions", "retest". Good keyword coverage, but a few natural variations are missing (e.g., "eval", "benchmark results", "test suite"), so it fits the 4 anchor ("Good keyword coverage; a few natural terms missing") rather than the comprehensive 5 anchor with synonyms and extensions.

4 / 5

Distinctiveness Conflict Risk

Has a clear niche — benchmark harness for the ToolUniverse/devtu ecosystem — with distinct triggers (benchmarks, regression check) and explicit positioning as the harness that routes fixes rather than applies them ("route fixes to devtu skills"). Minor overlap risk remains with closely related self-improvement skills (e.g., a devtu-self-evolve-style loop), fitting the 4 anchor rather than the minimal-conflict 5 anchor.

4 / 5

Total

17

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

referenced_paths_exist

Referenced path issues: 2 missing

Warning

Total

15

/

16

Passed

Repository

mims-harvard/ToolUniverse

Path

skills/devtu-benchmark-harness/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.