CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

eval-guide

Guide for running statistically meaningful agent-tty evals with trials, parallelism, and A/B comparison. Covers non-determinism baseline, recommended sample sizes, and result interpretation.

65

Quality

82%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

88%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

An excellent instruction-only body: concrete commands, quantified project-specific findings, a clearly sequenced A/B workflow with validation and feedback loops, and clean section organization. The only real gaps are minor redundancy around concurrency guidance and a somewhat long quick-reference section that could be split into a separate file.

DimensionReasoningScore

Conciseness

Largely efficient: dense bullets, bolded project-specific numbers ('~15-17% pass/fail flip rate', '3.4x wall-clock speedup', '23 of 24 cases inconclusive') that Claude could not know, with no explanation of generic concepts. Not 5 because of minor repetition — concurrency guidance appears in both sections 2 and 5, and section 5's closing line restates earlier bullets.

4 / 5

Actionability

Fully executable throughout: copy-paste CLI commands with all flags ('--trials 5 --concurrency 4 --compare-baseline "$BASELINE_JSON"'), concrete recommended trial counts per lane, explicit practical cutoffs ('0.05 score delta'), and a complete A/B loop including jq extraction of the baseline path. Covers prompt, execution, dogfood, and full-comparison cases.

5 / 5

Workflow Clarity

The before/after workflow is a clearly numbered 5-step sequence with explicit checkpoints (save the baseline report.json path, run candidate with --compare-baseline, read verdicts), plus feedback loops: interpretation rules ('do not call something better just because the mean moved'), the inconclusive-is-healthy default, and 'add more trials before making a claim' when results are noise-dominated. Section 7 adds a rerun-'npm run test' validation step for new cases.

5 / 5

Progressive Disclosure

No bundle files exist; the skill is a single well-organized file with ten clearly numbered sections and no nested or dangling references. Not 5 because at ~205 lines some reference-grade material (the six-command quick-reference cookbook in section 6, the authoring API in section 7) is inlined where a one-level-deep references file would keep the overview leaner.

4 / 5

Total

18

/

20

Passed

Description

66%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A specific, well-scoped description with strong domain keywords, but it answers only 'what' and never 'when' — there is no 'Use when...' or equivalent trigger clause, which caps completeness and slightly raises conflict risk. Adding an explicit trigger sentence would lift it from good to excellent.

Suggestions

Append an explicit trigger clause, e.g. 'Use when evaluating whether a skill or prompt change actually helped, when running agent-tty eval lanes, or when comparing eval results against a baseline.'

Add the natural trigger phrases users would actually say — 'evaluate a skill change', 'A/B test a prompt', 'benchmark before and after' — to improve trigger_term_quality and distinctiveness.

Consider mentioning the paired baseline comparison verdicts ('improved', 'regressed', 'inconclusive') since interpreting those outcomes is a primary use case.

DimensionReasoningScore

Specificity

Names the domain ('agent-tty evals') and several concrete capabilities: 'trials, parallelism, and A/B comparison', 'non-determinism baseline, recommended sample sizes, and result interpretation' — six specific items with minor gaps. Not 5 because the 'Guide for running...' framing is coverage-list rather than a comprehensive verb-led action list; not 3 because it far exceeds 1-2 concrete actions.

4 / 5

Completeness

The 'what' is clear (running statistically meaningful evals with trials, parallelism, A/B comparison), but there is no 'Use when...' clause or equivalent explicit trigger guidance, which caps completeness at 3 per the judging guidelines. Not 2 because the 'what' half is concrete and multi-part.

3 / 5

Trigger Term Quality

Good natural keyword coverage for the domain: 'evals', 'trials', 'A/B comparison', 'baseline', 'sample sizes' — phrases a user running evals would plausibly say. Not 5 because common variations like 'evaluate a skill change', 'benchmark', 'did my prompt change help', or flag/command names are missing.

4 / 5

Distinctiveness Conflict Risk

Scoped to a specific niche ('statistically meaningful agent-tty evals', 'trials', 'paired baseline comparison') making it mostly distinct from generic eval or testing skills. Not 5 because without explicit trigger phrases, generic requests about 'evaluating' or 'testing a change' could still pull it in over related skills.

4 / 5

Total

15

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

coder/agent-tty

Path

.mux/skills/eval-guide/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.