Builds a quarantine workflow for flaky tests - marks the test with the framework's skip/fixme/retry annotation, records the failure-rate observation and a bisect link in the annotation body, sets an auto-expiry date, and produces a CI report listing every quarantined test that has expired and needs re-evaluation. Use when a flaky test is blocking the trunk and must be removed from the gating path without losing track of it.
88
84%
Does it follow best practices?
Impact
89%
1.45xAverage score across 10 eval scenarios
Low
Low-risk findings worth noting
Compliance asked for a list of integration tests that do not execute, and I
could not produce one. Our Java suite uses @Disabled in some places, a
commented-out block in one place, and one test that is annotated but whose
reason is a person's name and nothing else.
I ran a git log over the file to get dates and the last commit that touched each of them, and I pulled the failure history for the ones that still run sometimes on the nightly job. Both are attached, along with what I could find out about the tickets referenced.
Today is 2026-08-17. The suite is JUnit 5.
What compliance actually wants is not the list, it is confidence that a test being off is a decision somebody made with an end in sight, and not something that happened. Right now I cannot demonstrate that for any of the five.
src/test/java/com/example/OrderServiceIT.java in the state you
would commit.docs/disabled-tests.md — the list compliance asked for, one row per
test that does not execute after your change, with enough per row that
somebody outside the team can see the decision and its end point.Extract the following files before beginning.
=============== FILE: src/test/java/com/example/OrderServiceIT.java =============== package com.example;
import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*;
class OrderServiceIT {
@Test
@Disabled
void refundPostsToLedger() {
var order = OrderFixtures.captured(4900);
var result = service.refund(order.id());
assertEquals("refunded", result.status());
assertEquals(-4900, ledger.lastEntry().amountCents());
}
@Test
@Disabled("flaky")
void bulkImportHandlesDuplicates() {
var report = service.bulkImport(Fixtures.csvWithDuplicates());
assertEquals(312, report.accepted());
assertEquals(4, report.rejectedDuplicates());
}
@Test
@Disabled("waiting on infra ticket #221")
void webhookRetriesOnGatewayTimeout() {
var delivery = service.deliverWebhook(Fixtures.orderCreated());
assertEquals(3, delivery.attempts());
}
// @Test
// void legacyXmlExportMatchesSchema() {
// var xml = service.exportLegacyXml(OrderFixtures.captured(4900));
// assertTrue(SchemaValidator.validate(xml, "orders-v2.xsd"));
// }
@Test
void orderTotalsIncludeTax() {
var order = OrderFixtures.captured(4900);
assertEquals(5390, service.totalWithTax(order).cents());
}}
=============== FILE: reports/audit-input.md ===============
| Test | Turned off | Last touched | Nightly runs | Failures | Rate |
|---|---|---|---|---|---|
| refundPostsToLedger | 2026-06-22 | 2026-06-22 | 90 (nightly) | 20 | 22.2% |
| bulkImportHandlesDuplicates | 2026-07-30 | 2026-07-30 | 90 (nightly) | 8 | 8.9% |
| webhookRetriesOnGatewayTimeout | 2026-04-02 | 2026-04-02 | 90 (nightly) | 0 | 0.0% |
| legacyXmlExportMatchesSchema | 2025-06-11 | 2025-06-11 | not run | — | — |
Notes gathered:
exportLegacyXml no longer exists on the
service.Owners: order/refund surfaces @web-platform (lead @kdavies); import and webhooks @integrations (lead @tobrien).