sq-ci-triage

Triage procedure for a red CI run on sq. Policy lives in AGENTS.md, especially Flaky tests and Git branch naming. How CI is wired is in docs/CI.md. This skill is the operational layer: what to check, in what order, and what to do with the answer.

It exists because of a measured failure mode. Between 2026-06-21 and 2026-09-16, thirteen autofix PRs were opened against this repo and none merged. references/case-file.md records each one and why it was closed. Four causes account for all thirteen: the fix was a t.Skip on a flake with a real root cause, the flake was already tracked or already being fixed, the diff carried work that belonged to another branch, or the PR's own green checks never ran the job that had failed.

Two rules follow from that, and they outrank convenience:

1. Never silence a failing test. Find the mechanism that makes it fail. The cause outlives the symptom, and it resurfaces against whatever test loses the race next.
2. Reporting is a real outcome. A good issue beats a speculative PR. Opening nothing beats opening a PR you cannot defend.

Order of work

Do these in order. Every step before "Classify" is a possible stop, and stopping early is cheap.

1. Symptom

Read the log. Not the check name, the log.

gh run view <run-id>                       # job list and conclusions
gh run view --job <job-id> --log-failed    # the failing step's output

Write down three things before going further:

• The failing job name, exactly (test-windows-full, test-nix (macos-15), run / test oracle:latest).
• The failing test's full name including subtest path, or the workflow step name when no test is involved.
• The verbatim error line. The mechanism is usually named in it, and it is usually unrelated to what the test name suggests. The process cannot access the file because it is being used by another process is a Windows sharing violation, whatever test it lands on.

One symptom per pass. If several tests failed, group them by verbatim error first: failures that share an error usually share a mechanism, and they are one symptom, not several. #1197 and #1200 were two tests failing from one DuckDB fixture lock. Take one group and leave the rest. Multiple symptoms in one PR cannot be reviewed or reverted independently.

2. Prior art

Search before you think about a fix. Every recent flake on this repo already had an issue, an open PR, or a prior attempt by the same automation, and the search that finds it takes seconds.

gh search issues --repo neilotoole/sq --include-prs --limit 20 'TestStartMemStatsTracker'
gh search issues --repo neilotoole/sq --include-prs --limit 20 'cmd_sql_test.go'
gh search issues --repo neilotoole/sq --include-prs --limit 20 'sakila.duckdb'
gh pr list --repo neilotoole/sq --state open --limit 40 \
  --json number,title,headRefName,files

Search by identifier, not by prose, and run more than one search. Error text does not match: a search for sakila.duckdb sharing violation returns nothing. Identifiers do, but which one hits varies. At the moment PR #1200 was opened, its own test name returned nothing that owned the failure, while sakila.duckdb and cmd_sql_test.go each returned the owning issue #1198 and the merged fix #1199. Search the test name, the failing file's path, the fixture filename, and the symbol named in the error.

Check all four:

• An open or recently closed issue describing the symptom. If it exists, it owns the problem.
• An open PR touching the files the failure implicates, including the test harness (testh/, testh/tu/) rather than only the failing test. A harness fix in flight will resolve failures in tests that look unrelated to it.
• A merged PR from the last few days that already fixed it. A nightly run can be older than its own fix, and a failure on a stale head is not a failure.
• Your own open and recent PRs. Three separate PRs went after the same downloader flake because nobody looked.

If any of those own the symptom, stop. Post a comment only if you carry a genuinely new data point, such as a run link for an occurrence on a platform not yet listed, and never repeat what the thread already says.

Count your own open PRs before continuing. Three or more open means stop and do Clean up instead: on 2026-09-15 seven autofix PRs were open at once, and all seven were closed unmerged.

3. Provenance

Establish what was actually tested, because it decides where a fix can go.

gh run view <run-id> --json headBranch,headSha,event,createdAt

• Failure on master or a nightly or dispatch run of master. The base for any fix is master.
• Failure on a pull request's branch. The problem belongs to that PR. If the PR is yours, push the fix to its branch. Otherwise comment on the PR with your findings, and do not push to someone else's branch. Do not open a competing PR against master. That was the single largest source of unmergeable diffs here: PR #1197 was 517 added lines against master when the change it intended was one line, because it branched from the head of another open PR and carried that PR's work along with it.
• Failure on a commit that is no longer the head. Re-check on current head before doing anything. If it no longer reproduces there, stop.

For a master failure, branch from current origin/master once the failure reproduces there, never from another feature branch:

git fetch origin master && git checkout -b fix/gh<issue>-<short-desc> origin/master

For a failure on your own PR, work on that PR's branch; there is no new branch to create.

4. Classify

"Intermittent with the mechanism established" is a narrow category. It means an explanation that predicts the observed pattern: why this OS, why this pair of tests, why now. If your explanation does not predict which runs fail, it is a guess, and a guess belongs in an issue.

Never silence a failure

These are not fixes, and a PR containing one will be closed:

• t.Skip, t.Skipf, tu.SkipWindows, tu.SkipIf or any conditional skip added because a test sometimes fails.
• Deleting or commenting out a test, a subtest, or an assertion.
• Raising a timeout, loosening a threshold, or adding retries so a flake surfaces less often, without an established mechanism. A timeout that is itself the mechanism is a fix, like the completion timeout #1144 added for the #595 flake; say so and show it.
• Moving a test behind an envar that no CI leg sets, which is a skip with extra steps.
• Excluding a job or a package from a workflow.

Gating on a real precondition is different, and these are correct because each one states what the test requires:

tu.SkipIssue and tu.SkipIssueWindows do exist for a tracked, accepted skip. They are a maintainer's call, not yours. Propose one on the issue and wait; do not open the PR.

5. Proof

Reproduce before fixing, and demonstrate after. If you cannot do the first, you are reporting, not fixing.

go test ./path/to/pkg -run '^TestName$' -count=1 -v
go test ./path/to/pkg -run '^TestName$' -count=20   # for an intermittent failure
go test ./... -short                                # the PR dev loop

A pipe hides the exit code: go test ./... | tail reports success on a failing suite. Capture the status first. Server-backed driver tests skip silently without their SQ_TEST_SRC__* envar, so "passes locally" can mean "never ran".

Green PR checks are not proof

A pull request runs the fast loop only: lint, test-nix with -short on Linux and macOS, test-windows-smoke, format, dependency review, plus the touched driver's bookend legs. These do not run on a PR:

• test-windows-full, the full Windows suite, which is where most of this repo's flakes appear.
• The DB matrix beyond the touched driver's bookends.
• Coverage, CodeQL, and the nightly live-network test (SQ_TEST_NETWORK, gh #1158).

PRs #1197 and #1200 both showed every check green while test-windows-full, the job that had failed, was skipped. Neither change was verified by anything.

To run the full suite on your branch, dispatch Main Pipeline on it. workflow_dispatch sets FULL_RUN and enables test-windows-full:

gh workflow run main.yml --ref <your-branch>
gh run list --workflow main.yml --branch <your-branch> --limit 5

Link each completed run. For an intermittent failure, dispatch at least three times, five if the failure rate looked below half, and link them all. Wait for each run to finish before dispatching the next: Main Pipeline cancels in-progress runs on non-master branches.

Main Pipeline dispatch only proves failures in Main Pipeline. Two others need a different run:

• A DB matrix leg (run / test <engine>:<tag>). Dispatch DB integration with the failing engine and tag. Each engine input takes tags or latest, oldest, bookends or all. For a change that could affect every engine, dispatch db-scheduled.yml with -f mode=bookends.

  gh workflow run db-integration.yml --ref <your-branch> -f oracle=latest

• The live-network test. SQ_TEST_NETWORK is set on the nightly schedule only, not on dispatch, so no run you can start will exercise it. A fix for it cannot be proven before merge. File an issue with the proposed fix rather than opening a PR.

Dispatch needs actions: write. If your token does not have it, or the workflow refuses, the fix is unproven: say so in your report, and prefer an issue with a proposed fix over a PR whose only evidence is a check set that never ran the failing job.

Last check before you believe your own fix: would the failing step have reached the line you changed? PR #1197 skipped a test that Windows CI already skipped, because no SQ_TEST_SRC__SAKILA_OR is set there. The failure was in the fixture copy that ran before it. The change could not have mattered.

6. Publish

An issue, when the mechanism is not established

This is the good outcome for a flake you cannot yet explain, and it is worth more than a skip. #1198 is the model: the verbatim error, a "Seen in" list of runs with dates and test names, the mechanism as far as it is known, a proposal, and acceptance criteria. Label it ci, add windows when it is Windows-only. Say plainly what is not yet known.

A PR, when you have a fix and proof

Every one of these is a hard requirement.

• Base is the branch that failed, per Provenance.

• Diff contains nothing but the fix. Check it and read the list:

  git diff --stat origin/<base>...HEAD

  If a file you did not deliberately change appears, your base is wrong. Fix the base. Do not ship it with an explanation.

• Size is proportionate. A CI fix is small. More than about 50 non-test lines, more than five files, or any change to go.sum, a lockfile, or site/ content that the failure did not implicate means stop and report instead. PR #1073 rewrote 2,465 lines of site/bun.lock to fix a broken documentation link.

• No drive-by anything. No dependency bumps, no reformatting, no unrelated doc edits. A stale netlify-cli bump swept into PR #1110 was still being cited when it was closed a month later.

• Branch name follows AGENTS.md: fix/ for a bug fix, chore/ for CI and tooling, plus gh<ISSUE>- when an issue exists. Not feature/ for a fix, and not an opaque automation suffix.

• CHANGELOG.md gets an entry under ## Unreleased for a user-visible fix. A test-only or CI-only change does not.

• make fmt && make lint before pushing, always.

• PR body states the failing run link, the mechanism, why this fix addresses that mechanism, and the dispatch runs that demonstrate it. If a related issue exists, link it rather than restating it.

• Re-check prior art immediately before opening. Minutes matter here: PR #1145 was opened ten minutes after the PR that superseded it, while that PR was still open, and that PR merged nine minutes later.

7. Clean up

Leftover branches and stale PRs are part of the cost being measured. Seven orphaned branches from this automation had to be deleted by hand.

• Abandoning a line of work means deleting the branch: git push origin --delete <branch>.
• If your PR is superseded or made obsolete, close it yourself, say in one line what supersedes it, and delete the branch. Do not wait to be told.
• Never leave a pushed branch with no PR and no explanation.