flaky-test-detector

Identifies non-deterministic or unreliable tests through static code analysis and test result analysis. Use when Claude needs to find flaky tests, analyze test reliability, or investigate intermittent test failures. Supports Python (pytest, unittest) and Java (JUnit, TestNG) test frameworks. Trigger when users mention "flaky tests", "intermittent failures", "non-deterministic tests", "unreliable tests", or ask to "find flaky tests", "analyze test stability", or "why tests fail randomly".

1.04x

Quality

80%

Does it follow best practices?

Impact

97%

1.04x

Average score across 3 eval scenarios

Securityby

Passed

No findings from the security scan

Quality

Content

61%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is well-organized and actionable with real bundle files referenced appropriately, but it is held back by redundant explanatory sections and missing validation checkpoints in its batch-oriented workflow.

Suggestions

Tighten 'What Makes Tests Flaky' and the per-framework 'Common issues'/'Best practices' lists — Claude already knows these causes; reference flaky-patterns.md instead of restating them inline to improve conciseness.

Add an explicit validation checkpoint in the workflow (e.g., after running analyze_test_results.py, verify the output is well-formed and spot-check high-scoring tests before reporting) to satisfy the batch-operation feedback-loop requirement.

Move the inline 'Common patterns to detect' catalog into references/flaky-patterns.md and link to it, leaving SKILL.md as a leaner overview that earns a higher progressive-disclosure score.

Dimension	Reasoning	Score
Conciseness	Mostly efficient, but the 'What Makes Tests Flaky' section explains causes Claude already knows, and the per-framework 'Common issues'/'Best practices' lists partially duplicate the inline pattern catalog, adding padding.	3 / 5
Actionability	Provides executable script invocation ('python scripts/analyze_test_results.py test_results.json'), a concrete JSON input schema, specific API patterns to search for, and a copy-ready report example; minor gaps since fix code lives in referenced files.	4 / 5
Workflow Clarity	A clear 5-step workflow is present, but batch operations (scanning all test files, running the analysis script) lack validation checkpoints or feedback loops (e.g., verifying script output before reporting), capping the score at 3 per the destructive/batch guidance.	3 / 5
Progressive Disclosure	Good structure with one-level-deep references to real bundle files (flaky-patterns.md, remediation-strategies.md, analyze_test_results.py) clearly signaled; some pattern detail that could live in references is inlined, keeping it just below 5.	4 / 5
	Total	14 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is exemplary: it states concrete capabilities, names supported frameworks, and provides comprehensive natural trigger terms covering what and when explicitly. It uses third person throughout and avoids over-claims.

Dimension	Reasoning	Score
Specificity	Names the domain plus concrete actions ('static code analysis and test result analysis') and supported frameworks ('pytest, unittest' and 'JUnit, TestNG'), giving comprehensive coverage of specific capabilities.	5 / 5
Completeness	Explicitly answers both 'what' (identifies non-deterministic tests via two analysis methods) and 'when' ('Use when Claude needs...', 'Trigger when users mention...') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Comprehensive natural trigger terms including synonyms and variations — 'flaky tests', 'intermittent failures', 'non-deterministic tests', 'unreliable tests', 'analyze test stability', 'why tests fail randomly'.	5 / 5
Distinctiveness Conflict Risk	Clear niche (flaky-test detection) with distinct triggers and named frameworks, making overlap with unrelated skills minimal.	5 / 5
	Total	20 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: ArabelaTso/Skills-4-SE
Path: skills/flaky-test-detector/SKILL.md
Commit: 4f38503

Reviewed: 7 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.