CtrlK
BlogDocsLog inGet started
Tessl Logo

testland/flaky-test-quarantine

Builds a quarantine workflow for flaky tests - marks the test with the framework's skip/fixme/retry annotation, records the failure-rate observation and a bisect link in the annotation body, sets an auto-expiry date, and produces a CI report listing every quarantined test that has expired and needs re-evaluation. Use when a flaky test is blocking the trunk and must be removed from the gating path without losing track of it.

88

1.45x
Quality

84%

Does it follow best practices?

Impact

89%

1.45x

Average score across 10 eval scenarios

SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

task.mdevals/scenario-4/

Auditing what our integration suite is actually not running

Problem Description

Compliance asked for a list of integration tests that do not execute, and I could not produce one. Our Java suite uses @Disabled in some places, a commented-out block in one place, and one test that is annotated but whose reason is a person's name and nothing else.

I ran a git log over the file to get dates and the last commit that touched each of them, and I pulled the failure history for the ones that still run sometimes on the nightly job. Both are attached, along with what I could find out about the tickets referenced.

Today is 2026-08-17. The suite is JUnit 5.

What compliance actually wants is not the list, it is confidence that a test being off is a decision somebody made with an end in sight, and not something that happened. Right now I cannot demonstrate that for any of the five.

Output Specification

  1. Produce src/test/java/com/example/OrderServiceIT.java in the state you would commit.
  2. Write docs/disabled-tests.md — the list compliance asked for, one row per test that does not execute after your change, with enough per row that somebody outside the team can see the decision and its end point.
  3. Anything you conclude should no longer be in the file must actually be removed from it, not left commented out.

Input Files

Extract the following files before beginning.

=============== FILE: src/test/java/com/example/OrderServiceIT.java =============== package com.example;

import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.Test; import static org.junit.jupiter.api.Assertions.*;

class OrderServiceIT {

@Test
@Disabled
void refundPostsToLedger() {
    var order = OrderFixtures.captured(4900);
    var result = service.refund(order.id());
    assertEquals("refunded", result.status());
    assertEquals(-4900, ledger.lastEntry().amountCents());
}

@Test
@Disabled("flaky")
void bulkImportHandlesDuplicates() {
    var report = service.bulkImport(Fixtures.csvWithDuplicates());
    assertEquals(312, report.accepted());
    assertEquals(4, report.rejectedDuplicates());
}

@Test
@Disabled("waiting on infra ticket #221")
void webhookRetriesOnGatewayTimeout() {
    var delivery = service.deliverWebhook(Fixtures.orderCreated());
    assertEquals(3, delivery.attempts());
}

// @Test
// void legacyXmlExportMatchesSchema() {
//     var xml = service.exportLegacyXml(OrderFixtures.captured(4900));
//     assertTrue(SchemaValidator.validate(xml, "orders-v2.xsd"));
// }

@Test
void orderTotalsIncludeTax() {
    var order = OrderFixtures.captured(4900);
    assertEquals(5390, service.totalWithTax(order).cents());
}

}

=============== FILE: reports/audit-input.md ===============

git log + nightly history for OrderServiceIT

TestTurned offLast touchedNightly runsFailuresRate
refundPostsToLedger2026-06-222026-06-2290 (nightly)2022.2%
bulkImportHandlesDuplicates2026-07-302026-07-3090 (nightly)88.9%
webhookRetriesOnGatewayTimeout2026-04-022026-04-0290 (nightly)00.0%
legacyXmlExportMatchesSchema2025-06-112025-06-11not run

Notes gathered:

  • refundPostsToLedger: turned off by @kdavies the day before the 5.3 freeze. No ticket was opened. The refund path is still shipped and still changing.
  • bulkImportHandlesDuplicates: @tobrien turned it off; the duplicate-detection race is understood but unfixed, ticket #5502 is open and unassigned.
  • webhookRetriesOnGatewayTimeout: infra ticket #221 was closed as done on 2026-05-14. Nightly has passed it 90 times since.
  • legacyXmlExportMatchesSchema: the legacy XML export endpoint was withdrawn in the 4.9 release (2025-08-30). exportLegacyXml no longer exists on the service.

Owners: order/refund surfaces @web-platform (lead @kdavies); import and webhooks @integrations (lead @tobrien).

SKILL.md

tile.json