CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

fixing-flaky-tests

Guides an agent through reproducing, root-causing, fixing, and validating flaky tests in the PostHog monorepo. Use when a test fails intermittently in CI but passes on rerun or locally, when `hogli ci:insights` or the debugging-ci-failures skill classifies a failure as a flaky test, when given a GitHub Actions URL for a flaky job, when asked to check Trunk Flaky Tests for a test, PR, or master, or when asked to deflake, stabilize, or fix a flaky Jest, pytest, or Playwright test. Core discipline: reproduce locally before changing anything, fix the root cause (never mask it with sleeps, retries, or bigger timeouts), and prove the fix with an N-run validation loop sized to the observed failure rate. Stabilizing is not the only valid outcome — the skill also gates whether the test should exist, so deleting a test that catches nothing real, or re-leveling one that flakes because of the level it runs at, are first-class endings.

Quality

93%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

—

The risk profile of this skill

SKILL.md
Quality
Evals
Security

Quality

Content

85%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A dense, highly actionable workflow with excellent sequencing, explicit validation loops, and repo-specific detail throughout — the body teaches nothing Claude already knows. Its one structural weakness is monolithic inlining: Trunk tooling reference and framework-specific cause catalogs belong in separate reference files, which would also shrink the main file.

Suggestions

Move the Trunk Flaky Tests material (collection short IDs, MCP tool parameters, auth instructions, CODEOWNERS projection details) into references/trunk.md and keep a short pointer plus the one command that matters in SKILL.md.

Split the frontend (MSW/kea) and backend (pytest) root-cause pattern catalogs into references/frontend-patterns.md and references/backend-patterns.md, linked from the symptom table, so the main file carries only the table and the bisect guidance.

Trim inline digressions such as the CODEOWNERS/owners.yaml projection explanation to a one-line pointer, keeping the main body focused on the workflow steps.

DimensionReasoningScore

Conciseness

Nearly every line is PostHog-specific operational fact Claude cannot know (commands, tool gotchas, statistical heuristics), with no generic-concept padding. However, ~310 lines with inlined reference-grade detail (Trunk collection IDs, CODEOWNERS projection mechanics) means some material could be trimmed or externalized — matching 'efficient; minor instances that could be trimmed' rather than 'every token earns its place'.

4 / 5

Actionability

Fully executable throughout: exact 'gh run list'/'gh api' commands, a complete N-run bash harness with exit-code judging and failure-log capture, 'git bisect run' invocations, CI-like jest flags, and a fill-in report template. Copy-paste-ready commands cover the common cases, including cost optimizations like building once and iterating with 'exec jest'.

5 / 5

Workflow Clarity

An 8-step numbered sequence with 'Three non-negotiables, in order', an escalation ladder whose stopping point becomes the validation environment, explicit validation sizing (N = max(3k, 20)), a feedback loop ('Any failure in the loop → back to step 4'), and a Boundaries section. Clear sequence with explicit validation checkpoints and error-recovery loops.

5 / 5

Progressive Disclosure

No bundle files exist, so all reference-grade material — the ~20-line Trunk tool reference (collection short IDs, MCP tool parameters, auth setup, CODEOWNERS projection) and the per-framework root-cause catalogs — is inlined in a single ~310-line SKILL.md. This matches 'content that should be separate is inline'; the under-50-line well-organized exception does not apply, and the inlined tool reference keeps it below 4.

3 / 5

Total

17

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is exemplary: concrete actions, explicit and varied trigger conditions, a clearly bounded niche, and an explicit handoff boundary with the adjacent CI-debugging skill. Dense but not padded — every clause adds a trigger or a capability.

DimensionReasoningScore

Specificity

Lists multiple concrete actions — 'reproducing, root-causing, fixing, and validating flaky tests' — plus first-class alternate outcomes (deleting, re-leveling), giving comprehensive coverage of the domain in third person. No gaps in coverage warrant a 4.

5 / 5

Completeness

Explicitly answers both what ('Guides an agent through reproducing, root-causing, fixing, and validating flaky tests in the PostHog monorepo') and when (an explicit 'Use when...' clause enumerating five concrete trigger scenarios). This matches the anchor with concrete trigger phrases, not the 4 where 'when' could be more specific.

5 / 5

Trigger Term Quality

Covers natural phrases users would say — 'fails intermittently in CI but passes on rerun', 'deflake, stabilize, or fix a flaky Jest, pytest, or Playwright test', 'GitHub Actions URL', 'Trunk Flaky Tests', 'hogli ci:insights'. Synonyms, framework names, and tool names are all present, so no common variation is missing.

5 / 5

Distinctiveness Conflict Risk

Clear niche (flaky tests in the PostHog monorepo) with distinct triggers, and it explicitly delineates the boundary with the sibling 'debugging-ci-failures' skill, minimizing overlap risk. Even closely related skills are handed off explicitly.

5 / 5

Total

20

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

relative_links

Relative link issues: 1 suspicious

Warning

Total

15

/

16

Passed

Repository

PostHog/posthog

Path

.agents/skills/fixing-flaky-tests/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.