CtrlK
BlogDocsLog inGet started
Tessl Logo

devtu-benchmark-harness

Continuous improvement system for ToolUniverse tools, skills, and plugin. Run benchmarks, diagnose failures, route fixes to devtu skills, retest. Use after skill optimization, tool additions, or as regression check.

67

Quality

81%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

77%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A highly actionable, well-sequenced workflow with strong validation feedback loops and copy-paste commands throughout. It is somewhat verbose and inlines reference-grade detail (grader internals, plugin architecture, failure tables) that would better live in separate files, and its one reference file is uncited.

Suggestions

Move the Grader strategy list, Plugin Architecture, and Known Failure Patterns tables into reference files and link to them from the body to tighten SKILL.md and improve progressive disclosure.

Actually cite references/benchmark-guide.md from the body (e.g. under Available benchmarks) so the existing reference is discoverable rather than orphaned.

Trim explanatory asides Claude can infer (e.g. why the char budget matters for routing) to lift conciseness toward the lean anchor.

DimensionReasoningScore

Conciseness

Mostly efficient with copy-paste commands and tables, but it leans long — sections like the plugin architecture rationale, grader strategy enumeration, and known-failure tables explain context Claude could largely infer, and could be trimmed or offloaded to a reference.

2 / 3

Actionability

Every step ships executable bash/python commands with real flags and arguments (run_harness_loop.sh, run_eval.py, analyze_results.py --diagnose), plus concrete category→skill and diagnosis→action routing tables that are directly copy-paste ready.

3 / 3

Workflow Clarity

The 5-step feedback loop is explicitly sequenced, the orchestrated runner collapses steps 0–4 into one command, and risky/batch operations carry validation checkpoints — the anti-memorization guard must pass before edits, retest verifies fixes, and root-cause investigation distinguishes agent vs GT errors.

3 / 3

Progressive Disclosure

Structure is well-sectioned and the benchmark-guide reference is one level deep, but much detail that could live in references (grader internals, plugin architecture, known-failure tables, per-benchmark score history) is inlined in SKILL.md, and the only reference file (benchmark-guide.md) is not actually cited from the body.

2 / 3

Total

10

/

12

Passed

Description

85%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description that pairs concrete actions with an explicit use-trigger and a distinct niche, written in third person. Its only weakness is reliance on project-internal jargon (ToolUniverse/devtu) that limits natural trigger-term coverage for users outside the project.

Suggestions

Add a couple of plain-language trigger synonyms users might say before learning the project vocabulary (e.g. "Use when evaluating tool or skill quality, running evals, or checking for regressions").

Consider clarifying what "devtu skills" are in one short clause so the routing action is understandable without project context.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — "Run benchmarks, diagnose failures, route fixes to devtu skills, retest" — each a distinct, verifiable operation rather than vague language.

3 / 3

Completeness

Clearly states what it does (the four-step loop) and provides an explicit "Use after skill optimization, tool additions, or as regression check" trigger clause covering both what and when.

3 / 3

Trigger Term Quality

Includes some natural terms ("benchmarks", "regression check", "skill optimization") but the core vocabulary ("ToolUniverse", "devtu skills") is project-specific jargon a user would only say if already inside the project, and common variations are sparse.

2 / 3

Distinctiveness Conflict Risk

The ToolUniverse/devtu benchmarking niche is narrow and distinct; the trigger conditions (skill optimization, tool additions, regression) are unlikely to fire for unrelated skills.

3 / 3

Total

11

/

12

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

referenced_paths_exist

Referenced path issues: 2 missing

Warning

Total

15

/

16

Passed

Repository
mims-harvard/ToolUniverse
Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.