CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/flaky-test-quarantine

Builds a quarantine workflow for flaky tests - marks the test with the framework's skip/fixme/retry annotation, records the failure-rate observation and a bisect link in the annotation body, sets an auto-expiry date, and produces a CI report listing every quarantined test that has expired and needs re-evaluation. Use when a flaky test is blocking the trunk and must be removed from the gating path without losing track of it.

88

1.45x
Quality

84%

Does it follow best practices?

Impact

89%

1.45x

Average score across 10 eval scenarios

SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

task.mdevals/scenario-9/

PR author wants his own new test switched off so his PR can land

Problem Description

PR #5120 adds subscription proration. It has been open eleven days and the author is frustrated, with some cause: the end-to-end job on that branch has gone red 14 times and only 3 of those were his code.

He has asked for two tests to be taken out of the blocking path so the PR can merge today. One is a navigation test that has nothing to do with his change and has been unreliable since spring. The other is the test he wrote in this PR for the proration behaviour itself. His argument for the second one is that it is "the same kind of intermittent as everything else" and that he will fix it in a follow-up next sprint, and he has a point that he did not create the mess he is stuck in.

The branch history is attached, along with the two tests. The proration test has never passed twice in a row since it was written. The one time the whole job went green on this branch, that test passed on its second attempt.

I want him unblocked today if there is a legitimate way to do it. I do not want his feature landing behind a switched-off test that was written specifically to check that feature.

Output Specification

  1. Edit the supplied spec files for whatever your decision covers, and leave the rest exactly as it is. Do not delete a test.
  2. Write docs/pr-5120-decision.md — the answer to the author, covering both requests, and stating what has to be true before the PR merges.
  3. Write docs/skipped-tests.md for anything taken out of the blocking path.

Input Files

Extract the following files before beginning.

=============== FILE: reports/branch-history.md ===============

PR #5120 — e2e job, 11 days, 31 runs on the branch

TestRunsFailuresRateOn main before this PR
nav sidebar collapses on mobile31412.9%yes — 11.4% over 290 runs
subscription upgrade prorates311961.3%no — added by this PR
subscription cancel refunds prorata3100.0%no — added by this PR

Notes:

  • nav sidebar collapses on mobile has been intermittent on main since 2026-03-xx. Ticket #3980 open, owner @web-platform (lead @kdavies). Nobody has worked on it this quarter.
  • subscription upgrade prorates was written in this PR. It has never passed on two consecutive runs. Its failures are an assertion mismatch on the prorated amount: expected 12.33, received 12.34 on 11 of the 19, and 12.32 on the other 8. No timeouts, no network errors.
  • Author: @rmatthews. Billing surfaces are owned by @web-platform.

=============== FILE: tests/nav.spec.ts =============== import { test, expect } from '@playwright/test';

test('nav sidebar collapses on mobile', async ({ page }) => { await page.goto('/'); await page.setViewportSize({ width: 390, height: 844 }); await page.getByRole('button', { name: 'Menu' }).click(); await expect(page.getByRole('navigation')).toHaveAttribute('data-state', 'collapsed'); });

test('nav search opens on slash', async ({ page }) => { await page.goto('/'); await page.keyboard.press('/'); await expect(page.getByRole('searchbox')).toBeFocused(); });

=============== FILE: tests/subscriptions.spec.ts =============== import { test, expect } from '@playwright/test';

test('subscription upgrade prorates', async ({ page }) => { await page.goto('/billing/upgrade'); await page.getByRole('button', { name: 'Upgrade to Team' }).click(); await expect(page.getByTestId('prorated-amount')).toHaveText('12.33'); });

test('subscription cancel refunds prorata', async ({ page }) => { await page.goto('/billing/cancel'); await page.getByRole('button', { name: 'Cancel plan' }).click(); await expect(page.getByTestId('refund-amount')).toHaveText('37.67'); });

SKILL.md

tile.json