flaky-detector

Detect flaky tests from CI history and propose LLM-validated fixes via quarantine pull requests. Use to find flaky tests, analyze CI test stability, identify tests that flip pass/fail without code changes, or set up automated quarantine workflows. Supports any test framework that emits JUnit XML (pytest, unittest, JUnit, TestNG, Vitest, Jest with junit reporter). Trigger when users mention "flaky tests", "intermittent failures", "tests that randomly fail", "quarantine flaky tests", "CI flakiness", or ask to "find unreliable tests", "analyze CI history", "mark tests as flaky".

2.11x

Quality

100%

Does it follow best practices?

Impact

95%

2.11x

Average score across 2 eval scenarios

Securityby

Low

Low-risk findings worth noting

Quality

Content

100%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

Excellent content: concise, executable, with a well-sequenced workflow containing validation checkpoints and properly structured one-level-deep references.

Dimension	Reasoning	Score
Conciseness	Lean and well-organized body with no padding or explanation of concepts Claude already knows; every section earns its place.	5 / 5
Actionability	Provides copy-paste-ready install/verify commands and concrete invocations with flags, plus full inputs and outputs tables covering the common cases.	5 / 5
Workflow Clarity	Quick Start is an explicitly ordered 5-step sequence with a dry-run validation checkpoint before the destructive PR-open step and feedback guidance ('confirm they match before going further').	5 / 5
Progressive Disclosure	SKILL.md is a concise overview with two clearly signaled one-level-deep references (references/flaky-patterns.md, references/quarantine-workflow.md) that exist as real files with no nesting.	5 / 5
	Total	20 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, comprehensive description: concrete capabilities, rich natural trigger phrases, and explicit what/when guidance with minimal conflict risk.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions (detect flaky tests, validate LLM-suggested fixes via AST, open quarantine PRs) plus a comprehensive supported-framework list, matching the comprehensive-coverage anchor.	5 / 5
Completeness	Explicitly answers what ('Detect flaky tests... propose LLM-validated fixes via quarantine pull requests') and when ('Use to...', 'Trigger when users mention...') with concrete trigger phrases.	5 / 5
Trigger Term Quality	Includes numerous natural trigger phrases with synonyms ('flaky tests', 'intermittent failures', 'tests that randomly fail', 'CI flakiness', 'mark tests as flaky'), matching comprehensive coverage including synonyms.	5 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (flaky-test detection and quarantine) with distinct, specific triggers and minimal overlap risk with other skills.	5 / 5
	Total	20 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: golikovichev/flaky-detector-agent
Path: SKILL.md
Commit: 1f7aa1a

Reviewed: about 9 hours ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.