Builds a quarantine workflow for flaky tests - marks the test with the framework's skip/fixme/retry annotation, records the failure-rate observation and a bisect link in the annotation body, sets an auto-expiry date, and produces a CI report listing every quarantined test that has expired and needs re-evaluation. Use when a flaky test is blocking the trunk and must be removed from the gating path without losing track of it.
88
84%
Does it follow best practices?
Impact
89%
1.45xAverage score across 10 eval scenarios
Low
Low-risk findings worth noting
We keep a markdown file listing every test we have switched off, and the matching annotations live in the specs. It was working for about two months and then it stopped being maintained. There are five entries. At least one has been sitting there since December.
The reason it came up: an auditor asked us how many tests in our suite do not actually run, and we could not answer without reading the specs by hand. Two of the five have dates that have already gone by, one has no date at all, and one of them I am fairly sure was fixed weeks ago and we just never turned the test back on.
The proposal on the table from our engineering manager is to push every date out ninety days from today so the list is "clean" again, and then set a reminder. Today is 2026-08-17.
I would rather do this properly once. Some of these should probably not be on the list any more at all.
docs/skip-review-2026-08-17.md — a verdict for each of the five
entries with the reasoning, in the same order as the log.docs/skipped-tests.md so it reflects the state after your
decisions. Entries that should no longer be on it must not be on it.tests/e2e.spec.ts to match your decisions.Extract the following files before beginning.
=============== FILE: docs/skipped-tests.md ===============
| ID | Test | Off since | Re-check by | Times extended | Ticket | Owner |
|---|---|---|---|---|---|---|
| Q-01 | checkout applies regional tax | 2025-12-08 | 2026-03-08 | 2 | #3120 | |
| Q-02 | search returns paged results | 2026-07-28 | 2026-08-27 | 0 | #5501 | @search |
| Q-03 | notification digest sends | 2026-06-01 | 2026-07-01 | 1 | #4412 | @messaging |
| Q-04 | legacy csv import parses | 2026-03-14 | 0 | |||
| Q-05 | dashboard widgets load | 2026-07-20 | 2026-08-10 | 1 | #5388 | @web-platform |
Notes kept by hand:
=============== FILE: tests/e2e.spec.ts =============== import { test, expect } from '@playwright/test';
test('checkout applies regional tax', async ({ page }) => { test.fixme(true, 'Off since 2025-12-08 (#3120). Re-check by 2026-03-08.'); await page.goto('/cart'); await expect(page.getByTestId('tax-row')).toContainText('VAT'); });
test('search returns paged results', async ({ page }) => { test.fixme(true, 'Off since 2026-07-28 (#5501) - fails ~7% of runs; indexer race. Re-check by 2026-08-27. Owner: @search.'); await page.goto('/search?q=widget'); await expect(page.getByTestId('result-count')).toHaveText('42 results'); });
test('notification digest sends', async ({ page }) => { test.fixme(true, 'Off since 2026-06-01 (#4412). Re-check by 2026-07-01. Owner: @messaging.'); await page.goto('/settings/notifications'); await expect(page.getByText('Daily digest queued')).toBeVisible(); });
test('legacy csv import parses', async ({ page }) => { test.fixme(true, 'Off since 2026-03-14. Flaky.'); await page.goto('/import'); await page.setInputFiles('input[type=file]', 'fixtures/rows.csv'); await expect(page.getByText('312 rows imported')).toBeVisible(); });
test('dashboard widgets load', async ({ page }) => { test.fixme(true, 'Off since 2026-07-20 (#5388) - fails ~9% of runs. Re-check by 2026-08-10. Owner: @web-platform.'); await page.goto('/dashboard'); await expect(page.getByTestId('widget-grid')).toBeVisible(); });