CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

test-flakiness

Find flaky tests from CI logs — aggregates pass rates, spots intermittent failures, recommends quarantine. After multiple runs.

Quality

82%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable, with concrete commands, engine-specific patterns, and explicit statistical-confidence gates at the right checkpoints. Its weakest point is token efficiency: a padded conceptual opening and version-anchored detail dilute otherwise lean material.

Suggestions

Trim the opening paragraph: drop the definition of flaky tests and the rationale about teams ignoring red CI — Claude already knows this; keep only the skill's mandate (identify, classify cause, recommend quarantine vs fix).

Replace the version pin "as of gdUnit4 6.2.2; confirm them there for the installed version" with a single instruction to verify the skip-parameter convention in addons/gdUnit4/src/core/GdUnitTestSuiteScanner.gd, moving the version note to a maintenance/deprecated section if it must be kept.

Move the per-engine parsing formats and skip mechanisms (GdUnit4/NUnit/Unreal blocks in sections 3 and 5) into engine-specific reference files under references/, keeping one line per engine in SKILL.md to reduce context load when only one engine is relevant.

DimensionReasoningScore

Conciseness

The opening paragraph ("A flaky test is one that sometimes passes and sometimes fails... they train the team to ignore red CI runs, masking genuine failures") explains concepts Claude already knows, and version-pinned detail ("as of gdUnit4 6.2.2") adds time-sensitive material outside any 'deprecated' section — more than minor over-explanation, though the tables, commands, and thresholds are otherwise tight.

3 / 5

Actionability

Concrete grep patterns per engine and format ("Result={Success}" / "Result={Fail}", "<testcase name="), exact skip syntax ("[Ignore("flaky: [cause]")]" for NUnit, "func test_x(_do_skip := true, _skip_reason := ...)" for gdUnit4), numeric fail-rate thresholds, and a copy-paste report template make the guidance fully executable for the common cases.

5 / 5

Workflow Clarity

Seven clearly sequenced sections with explicit checkpoints: fewer than 3 runs forces "suspected" labeling and a request for more data, no-log-data stops and asks the user, and writes require approval. It falls short of a 5 because there is no post-edit validation of the regression-suite update (e.g., verifying the quarantine table still parses or that entries landed correctly).

4 / 5

Progressive Disclosure

A single, well-sectioned file with clear headers and one-level-deep pointers to project docs ("docs/engine-reference/unreal/current-best-practices.md", "addons/gdUnit4/src/core/GdUnitTestSuiteScanner.gd") and no nested references. It is not a 5 because engine-specific parsing and skip-mechanism detail is inlined in a 240-line skill where per-engine reference files would keep the overview leaner.

4 / 5

Total

16

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: third-person voice, concrete actions, and an explicit (if brief) when-clause. Its main weaknesses are the terseness of the trigger guidance and missing natural synonyms such as "unstable tests" or "flakiness".

DimensionReasoningScore

Specificity

"aggregates pass rates, spots intermittent failures, recommends quarantine" lists multiple concrete actions that comprehensively cover the skill's workflow (detect, diagnose, act), with no meaningful coverage gaps.

5 / 5

Completeness

"aggregates pass rates, spots intermittent failures, recommends quarantine" answers 'what' explicitly, and "After multiple runs." is an explicit when-clause, but the 'when' is terse and lacks concrete trigger phrases (e.g., 'when developers dismiss CI failures as flaky'), placing it below the 5 anchor and above the 3 anchor.

4 / 5

Trigger Term Quality

Natural keywords "flaky tests", "CI logs", "intermittent failures", and "quarantine" are present and would be said by users, but common synonyms/variants like "unstable tests" or "flakiness" are missing, so coverage is good rather than comprehensive.

4 / 5

Distinctiveness Conflict Risk

"Find flaky tests from CI logs" carves out a clear niche (statistical flakiness detection from run history) with distinct triggers, making it unlikely to fire for a wrong skill.

5 / 5

Total

18

/

20

Passed

Validation

87%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 14 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

allowed_tools_field

'allowed-tools' contains unusual tool name(s)

Warning

frontmatter_unknown_keys

Unknown frontmatter key(s) found; consider removing or moving to metadata

Warning

Total

14

/

16

Passed

Repository

Donchitos/Claude-Code-Game-Studios

Path

.claude/skills/test-flakiness/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.