devtu-benchmark-harness

Continuous improvement system for ToolUniverse tools, skills, and plugin. Run benchmarks, diagnose failures, route fixes to devtu skills, retest. Use after skill optimization, tool additions, or as regression check.

Quality

71%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./skills/devtu-benchmark-harness/SKILL.md

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with a clear 5-step workflow, validation guard, and good file structure, but it carries some inlined detail that would be better offloaded to references. Overall a strong, executable skill.

Suggestions

Move the full Grader strategy list and Known Failure Patterns table into a reference file (e.g. references/grader-and-patterns.md) and link to it, keeping only a one-line summary in SKILL.md to improve conciseness and progressive disclosure.

Add an explicit retest pass/fail checkpoint (e.g. 'only update baseline if wrong->correct flips exceed regressions') to strengthen validation feedback loops.

Tighten the Anti-memorization guard section by linking the flag definitions to check_memorization.py rather than restating them inline.

Dimension	Reasoning	Score
Conciseness	Mostly efficient with commands and tables doing heavy lifting, but sections like the Grader strategy list and Known Failure Patterns include detail that pads the body with material Claude could derive from the bundled scripts.	3 / 5
Actionability	Provides fully executable, copy-paste-ready commands at every step (run_harness_loop.sh, run_eval.py invocations with flags) plus concrete diagnosis-to-skill routing tables covering common cases.	5 / 5
Workflow Clarity	The 5-step loop is clearly sequenced with a diagnose-retest feedback loop and validation via check_memorization.py, though a few batch/retest steps could surface explicit pass/fail checkpoints more prominently.	4 / 5
Progressive Disclosure	Well-structured overview that points to one-level-deep bundle files (scripts/, references/benchmark-guide.md) with clear sectioning; minor gaps where some referenceable detail (grader internals, failure patterns) is inlined rather than linked out.	4 / 5
	Total	16 / 20 Passed

Description

67%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific and distinct, clearly conveying the continuous-improvement feedback loop and naming concrete actions. It is held back by trigger-term coverage and an only moderately explicit when-clause.

Suggestions

Broaden the 'Use when...' clause with concrete user-facing trigger phrases like 'after changing a tool', 'when benchmark scores drop', or 'as a regression check before release'.

Add natural-sounding synonyms users might say (e.g., 'evaluate', 'test the plugin', 'find failing questions') to improve trigger matching.

Keep the action list but ensure the phrasing stays in third person to avoid any voice penalty.

Dimension	Reasoning	Score
Specificity	Lists several concrete actions ("Run benchmarks, diagnose failures, route fixes to devtu skills, retest") tied to a named system, with minor gaps in coverage of the full improvement scope.	4 / 5
Completeness	Clearly states what it does and provides a "Use after..." trigger clause, though the when-guidance is scoped to a narrow set of situations rather than comprehensive concrete trigger phrases.	4 / 5
Trigger Term Quality	Includes some relevant terms ("benchmarks", "regression check", "skill optimization", "tool additions") but misses the natural variations and synonyms a user would commonly say when invoking this skill.	3 / 5
Distinctiveness Conflict Risk	Targets a clear niche (benchmark-driven improvement of the ToolUniverse plugin) with mostly distinct triggers, with only minor overlap risk against related devtu skills.	4 / 5
	Total	15 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
referenced_paths_exist	Referenced path issues: 2 missing	Warning

	Total	15 / 16 Passed

Repository: mims-harvard/ToolUniverse
Path: skills/devtu-benchmark-harness/SKILL.md
Commit: 1b02920

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.