bat-story-eval

Compare MCP tool behavior between target and baseline versions using pre-built and custom stories with diff-based triage.

Quality

80%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

High

Do not use without reviewing

Fix and improve this skill with Tessl

tessl review fix ./.claude/skills/bat-story-eval/SKILL.md

Quality

Content

92%Weight 40%Scale 1-3

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The SKILL.md body is a lean, highly actionable, well-sequenced evaluation workflow with concrete commands and verification checkpoints throughout. Its only real gap is progressive disclosure: it ships reference files that are never linked or signaled from the body.

Suggestions

Link the existing bundle files from the body — e.g., in the scoring section add 'See [evaluation-protocol.md](references/evaluation-protocol.md) for the full black-box/white-box scoring criteria' and '[regression-protocol.md](references/regression-protocol.md) for regression triage' — so the reference files are discoverable.

Consider moving the detailed scoring matrix and token-extraction formulas into the referenced protocols and keeping the body as an overview, to better match the progressive-disclosure intent.

Dimension	Reasoning	Score
Conciseness	The body is dense and actionable with no padding explaining concepts Claude already knows (MCP, Home Assistant, git diff); nearly every token earns its place in commands, YAML formats, and scoring matrices.	3 / 3
Actionability	Provides fully executable bash commands, complete Python snippets for token extraction, an exact custom-story YAML template, and a concrete scoring matrix — copy-paste ready with specific file paths and flags.	3 / 3
Workflow Clarity	Steps 0–8 are explicitly ordered ('Follow these steps IN ORDER. Do not skip steps') with verification checkpoints after each story ('ALWAYS verify ... before running the next') and feedback loops for regressions (re-run to check flakiness, white-box analysis).	3 / 3
Progressive Disclosure	Bundle files exist (references/evaluation-protocol.md, references/regression-protocol.md) but the body never signals or links to them, leaving them orphaned; relevant protocol detail is either inline or unreachable rather than cleanly split with one-level-deep navigation.	2 / 3
	Total	11 / 12 Passed

Description

67%Weight 40%Scale 1-3

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific and distinctive about a well-scoped niche (MCP tool version comparison with diff-based triage), but it omits an explicit 'when to use' trigger clause and relies on technical jargon over natural user-facing keywords.

Suggestions

Add an explicit 'Use when ...' clause naming natural triggers such as 'Use when comparing agent/MCP tool behavior between a released version and your local changes, or when checking for regressions after a ha-mcp diff.'

Surface natural keywords users would actually say (e.g., 'regression', 'compare versions', 'agent behavior', 'UAT') alongside the existing technical terms.

Dimension	Reasoning	Score
Specificity	Names the concrete domain and actions: 'Compare MCP tool behavior between target and baseline versions using pre-built and custom stories with diff-based triage' lists multiple specific operations (compare versions, pre-built/custom stories, diff-based triage).	3 / 3
Completeness	Clearly states what the skill does but provides no explicit 'Use when...' trigger clause; per the judging guideline, a missing explicit trigger caps completeness at 2.	2 / 3
Trigger Term Quality	Includes relevant technical keywords ('MCP tool behavior', 'baseline versions', 'diff-based triage') but lacks common natural-language variations a user would say such as 'regression test', 'compare versions', or 'agent behavior'.	2 / 3
Distinctiveness Conflict Risk	The niche is highly specific — comparing MCP tool behavior across git versions with diff-based triage — making it unlikely to trigger for or conflict with unrelated skills.	3 / 3
	Total	10 / 12 Passed

Validation

87%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

Criteria	Description	Result
allowed_tools_field	'allowed-tools' contains unusual tool name(s)	Warning
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	14 / 16 Passed

Repository: homeassistant-ai/ha-mcp
Path: .claude/skills/bat-story-eval/SKILL.md
Commit: 01964cc

Reviewed: 11 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.