Name: fixing-flaky-tests
Rating: 72.8 (1 reviews)
Author: PostHog

fixing-flaky-tests

Guides an agent through reproducing, root-causing, fixing, and validating flaky tests in the PostHog monorepo. Use when a test fails intermittently in CI but passes on rerun or locally, when `hogli ci:insights` or the debugging-ci-failures skill classifies a failure as a flaky test, when given a GitHub Actions URL for a flaky job, or when asked to deflake, stabilize, or fix a flaky Jest, pytest, or Playwright test. Core discipline: reproduce locally before changing anything, fix the root cause (never mask it with sleeps, retries, or bigger timeouts), and prove the fix with an N-run validation loop sized to the observed failure rate. Stabilizing is not the only valid outcome — the skill also gates whether the test should exist, so deleting a test that catches nothing real, or re-leveling one that flakes because of the level it runs at, are first-class endings.

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Quality

Content

85%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A dense, highly actionable workflow with strong sequencing and validation feedback loops, assuming Claude's competence throughout. Its main weakness is progressive disclosure: everything lives in one large SKILL.md with no bundled reference files to offload the pattern catalogs and detailed tables.

Suggestions

Move the Frontend (Jest + kea) and Backend (pytest) PostHog-specific pattern catalogs into a `references/posthog-flake-patterns.md` file, keeping only a one-line pointer and the most common example inline in SKILL.md.

Extract the detailed N-run loop harness, sizing math (N ≥ 3k), and cost notes into a `references/validation-loop.md` referenced from step 7, so the main body stays a navigable overview.

Consider a `references/symptom-cause-table.md` for the step-4 symptom→cause class and masking-move tables, signaling them as one-level-deep references to reduce the monolithic wall of tables.

Dimension	Reasoning	Score
Conciseness	Lean, high-signal prose that assumes Claude's competence (symptom/cause tables, bisect caveats, loop cost notes) with no concept tutorials, but a few explanatory asides (the `p^30 ≈ 0` math rationale, the build-rebuild note) slightly exceed 'every token earns its place'.	4 / 5
Actionability	Fully executable, copy-paste-ready commands throughout — `gh run list`, the `gh api .../logs \| grep` shard check, the complete N-run bash loop harness, `git bisect run bash -c '...'`, and the filled-in report template — covering the common cases as in the anchor-5 example.	5 / 5
Workflow Clarity	A clear 8-step sequence with explicit validation checkpoints and feedback loops ('Any failure in the loop → back to step 4', the step-3 escalation 'Stop at the first level that reproduces it'), plus an approval+validation gate around the destructive deletion outcome — matching the anchor-5 pattern.	5 / 5
Progressive Disclosure	Good section structure and tables, but the skill is a ~250-line monolithic SKILL.md with no bundle files and no signaled one-level-deep reference files; substantial catalogs (frontend/backend patterns, bisect, symptom tables) that could live in separate references are inlined, fitting the anchor-3 'content that should be separate is inline'.	3 / 5
	Total	17 / 20 Passed

Description

92%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A highly specific, well-triggered description that clearly states both what the skill does and when to invoke it, with strong disambiguation from neighboring skills. The only minor weakness is reliance on internal tool jargon (`hogli ci:insights`) among the trigger terms.

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions ('reproducing, root-causing, fixing, and validating') plus concrete discipline ('N-run validation loop sized to the observed failure rate') and outcomes ('deleting', 're-leveling') — comprehensive coverage matching the anchor-5 example.	5 / 5
Completeness	Explicitly answers 'what' ('Guides an agent through reproducing, root-causing, fixing, and validating flaky tests') and 'when' ('Use when a test fails intermittently in CI but passes on rerun or locally...') with concrete trigger phrases — both halves clearly present as in the anchor-5 example.	5 / 5
Trigger Term Quality	Strong natural phrasing ('test fails intermittently in CI but passes on rerun', 'deflake, stabilize, or fix a flaky Jest, pytest, or Playwright test') with synonyms and tool names, but `hogli ci:insights` is internal jargon rather than a phrase a user would naturally say, keeping it just below comprehensive.	4 / 5
Distinctiveness Conflict Risk	Clear niche (flaky tests in the PostHog monorepo) with explicit disambiguation from sibling skills ('use the debugging-ci-failures skill first... this skill takes over once the failure is classified as a flaky test'), giving minimal conflict risk.	5 / 5
	Total	19 / 20 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: PostHog/posthog
Path: .agents/skills/fixing-flaky-tests/SKILL.md
Commit: 06292db

Reviewed: 22 minutes ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.