CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

fixing-flaky-tests

Guides an agent through reproducing, root-causing, fixing, and validating flaky tests in the PostHog monorepo. Use when a test fails intermittently in CI but passes on rerun or locally, when `hogli ci:insights` or the debugging-ci-failures skill classifies a failure as a flaky test, when given a GitHub Actions URL for a flaky job, when asked to check Trunk Flaky Tests for a test, PR, or master, or when asked to deflake, stabilize, or fix a flaky Jest, pytest, or Playwright test. Core discipline: reproduce locally before changing anything, fix the root cause (never mask it with sleeps, retries, or bigger timeouts), and prove the fix with an N-run validation loop sized to the observed failure rate. Stabilizing is not the only valid outcome — the skill also gates whether the test should exist, so deleting a test that catches nothing real, or re-leveling one that flakes because of the level it runs at, are first-class endings.

Quality

94%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

The canonical home for this skill is fixing-flaky-tests in PostHog/posthog

SKILL.md
Quality
Evals
Security

Quality

Content

88%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A dense, highly actionable workflow skill that front-loads three non-negotiables, sequences an 8-step process with explicit validation and feedback loops, and is rich in genuinely non-obvious repo-specific knowledge (tooling caveats, cost notes, statistical sizing). Its two weaknesses are modest: minor rhetorical padding in the decision-framing prose, and a monolithic ~290-line body whose PostHog-specific pattern catalogs could be split into a reference file.

Suggestions

Move the frontend (Jest/kea/MSW) and backend (pytest) root-cause pattern catalogs out of step 4 into a references/ file (e.g. references/posthog-flake-patterns.md), keeping a symptom-to-cause table inline so the main body stays a navigable overview.

Trim the persuasive framing in step 5 (e.g. "the one case where cost is already proven and value is not") and condense the broken-tests/test-health surface taxonomy into a two-line do/don't — the operational rules already carry the point.

The git bisect usage explanation is close to standard knowledge; keep only the flake-specific caveat (lucky pass per step, or use the CI run-history boundary) and drop the basic start/run commands.

DimensionReasoningScore

Conciseness

Nearly all content is non-obvious environment-specific fact Claude cannot know (MSW unhandled-request behavior, LISTENER_FINISH_WAIT_TIMEOUT, ci:insights lacking a denominator, the pnpm build:products rebuild cost, the N ≥ 3k sizing math), so it earns its tokens. A few stretches could be trimmed — the rhetorical framing in step 5 ("the one case where cost is already proven and value is not") and the extended broken-tests/test-health taxonomy — keeping it below the every-token-earns-its-place anchor.

4 / 5

Actionability

Fully executable throughout: gh run list/gh api commands with real flags and --json fields, a complete copy-paste bash loop harness judging by exit code, git bisect run with a concrete repro wrapper, the pnpm exec jest variant that skips the rebuild, and a fill-in report template. Placeholders (<test command>, <run-id>) are the only substitutions needed.

5 / 5

Workflow Clarity

Eight numbered steps with an explicit five-level reproduction escalation ladder, a validation checkpoint in step 7 sized from the measured pre-fix rate (N = max(3k, 20)), an explicit feedback loop ("Any failure in the loop → back to step 4"), a three-outcome decision table, and hard safety gates — deletion and quarantine require explicit user approval. Validation is present at every risky juncture.

5 / 5

Progressive Disclosure

Well-sectioned file with clear headers, and external pointers are explicitly signaled with paths and what each contains (docs/internal/ci-things-already-tried.md, /writing-tests, frontend/src/mocks/jest.ts) — all one level deep, none nested. However, it is a ~290-line single file: the PostHog-specific frontend/backend root-cause pattern catalogs (kea/MSW/pytest details) are inline content that could live in a references/ file to slim the main body, which keeps it below the appropriately-split anchor.

4 / 5

Total

18

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: concrete third-person capabilities, an explicit multi-clause Use-when covering both natural-language and tool-specific triggers, comprehensive synonym coverage (deflake/stabilize/flaky), and a clearly delineated niche that names its own handoff boundary with the general CI-failure skill. No padding despite its length; every clause carries trigger or capability information.

DimensionReasoningScore

Specificity

Names four concrete actions ("reproducing, root-causing, fixing, and validating flaky tests") plus two additional outcomes ("deleting a test that catches nothing real, or re-leveling one"), all in third-person voice. Coverage is comprehensive with no generic filler.

5 / 5

Completeness

Explicitly answers both questions: the "what" is the four-phase workflow (reproduce, root-cause, fix, validate) and the outcome gate, and the "when" is a multi-clause "Use when..." with concrete trigger phrases. Neither is vague or merely implied.

5 / 5

Trigger Term Quality

Triggers span natural phrasings users would actually say: "test fails intermittently in CI but passes on rerun", "GitHub Actions URL for a flaky job", "deflake, stabilize, or fix a flaky Jest, pytest, or Playwright test", and tool-specific "Trunk Flaky Tests". Synonyms and framework names are all covered.

5 / 5

Distinctiveness Conflict Risk

Clear niche — flaky tests in the PostHog monorepo — with triggers keyed to flakiness specifically, and the boundary with the sibling debugging-ci-failures skill is itself stated as a trigger ("when ... the debugging-ci-failures skill classifies a failure as a flaky test"). Minimal overlap risk.

5 / 5

Total

20

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

relative_links

Relative link issues: 1 suspicious

Warning

Total

15

/

16

Passed

Repository

amplifthq/opentag

Path

.agents/skills/fixing-flaky-tests/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.