Builds a quarantine workflow for flaky tests - marks the test with the framework's skip/fixme/retry annotation, records the failure-rate observation and a bisect link in the annotation body, sets an auto-expiry date, and produces a CI report listing every quarantined test that has expired and needs re-evaluation. Use when a flaky test is blocking the trunk and must be removed from the gating path without losing track of it.
88
84%
Does it follow best practices?
Impact
89%
1.45xAverage score across 10 eval scenarios
Low
Low-risk findings worth noting
PR #5120 adds subscription proration. It has been open eleven days and the author is frustrated, with some cause: the end-to-end job on that branch has gone red 14 times and only 3 of those were his code.
He has asked for two tests to be taken out of the blocking path so the PR can merge today. One is a navigation test that has nothing to do with his change and has been unreliable since spring. The other is the test he wrote in this PR for the proration behaviour itself. His argument for the second one is that it is "the same kind of intermittent as everything else" and that he will fix it in a follow-up next sprint, and he has a point that he did not create the mess he is stuck in.
The branch history is attached, along with the two tests. The proration test has never passed twice in a row since it was written. The one time the whole job went green on this branch, that test passed on its second attempt.
I want him unblocked today if there is a legitimate way to do it. I do not want his feature landing behind a switched-off test that was written specifically to check that feature.
docs/pr-5120-decision.md — the answer to the author, covering both
requests, and stating what has to be true before the PR merges.docs/skipped-tests.md for anything taken out of the blocking path.Extract the following files before beginning.
=============== FILE: reports/branch-history.md ===============
| Test | Runs | Failures | Rate | On main before this PR |
|---|---|---|---|---|
| nav sidebar collapses on mobile | 31 | 4 | 12.9% | yes — 11.4% over 290 runs |
| subscription upgrade prorates | 31 | 19 | 61.3% | no — added by this PR |
| subscription cancel refunds prorata | 31 | 0 | 0.0% | no — added by this PR |
Notes:
nav sidebar collapses on mobile has been intermittent on main since
2026-03-xx. Ticket #3980 open, owner @web-platform (lead @kdavies). Nobody
has worked on it this quarter.subscription upgrade prorates was written in this PR. It has never passed
on two consecutive runs. Its failures are an assertion mismatch on the
prorated amount: expected 12.33, received 12.34 on 11 of the 19, and 12.32 on
the other 8. No timeouts, no network errors.=============== FILE: tests/nav.spec.ts =============== import { test, expect } from '@playwright/test';
test('nav sidebar collapses on mobile', async ({ page }) => { await page.goto('/'); await page.setViewportSize({ width: 390, height: 844 }); await page.getByRole('button', { name: 'Menu' }).click(); await expect(page.getByRole('navigation')).toHaveAttribute('data-state', 'collapsed'); });
test('nav search opens on slash', async ({ page }) => { await page.goto('/'); await page.keyboard.press('/'); await expect(page.getByRole('searchbox')).toBeFocused(); });
=============== FILE: tests/subscriptions.spec.ts =============== import { test, expect } from '@playwright/test';
test('subscription upgrade prorates', async ({ page }) => { await page.goto('/billing/upgrade'); await page.getByRole('button', { name: 'Upgrade to Team' }).click(); await expect(page.getByTestId('prorated-amount')).toHaveText('12.33'); });
test('subscription cancel refunds prorata', async ({ page }) => { await page.goto('/billing/cancel'); await page.getByRole('button', { name: 'Cancel plan' }).click(); await expect(page.getByTestId('refund-amount')).toHaveText('37.67'); });