Buildkite Pipeline Failure Investigation

Investigate Buildkite pipeline failures systematically to identify root causes. This skill uses the Buildkite CLI (bk) as the primary interface, with REST API fallback.

Prerequisites — Buildkite CLI Authentication

The bk CLI is the preferred way to interact with Buildkite. Before investigating, ensure it is installed and authenticated.

Step 0: Check and Install the CLI

which bk || echo "bk CLI not installed"

If bk is not installed, install it:

brew tap buildkite/buildkite && brew install buildkite/buildkite/bk

Step 1: Check Authentication

bk auth status

If authenticated, you will see the org slug, token UUID, scopes, and user info. Proceed to investigation.

If not authenticated, you will see:

Error: you are not authenticated. Run bk auth login to authenticate, or run bk use to select a configured organization

In this case, stop and ask the user to authenticate:

Please run bk auth login in a terminal to authenticate with Buildkite via browser-based OAuth. This is a one-time setup similar to aws sso login. Once done, run bk auth switch mockserver to select the organization.

Do NOT attempt to run bk auth login yourself — it requires interactive browser OAuth that cannot be completed in a non-TTY terminal.

Step 2: Select Organization (if needed)

If bk auth status shows selected_org: "", the org hasn't been selected:

bk auth switch mockserver

Step 3: Get API Token for REST API Calls

When you need the REST API (for endpoints not covered by bk CLI):

TOKEN=$(bk auth token 2>/dev/null)
if [ -z "$TOKEN" ]; then
  echo "ERROR: bk auth token failed — run 'bk auth login' first"
  exit 1
fi
curl -sH "Authorization: Bearer $TOKEN" "https://api.buildkite.com/v2/..."

This extracts the OAuth token from the CLI's keychain storage. Always validate the token is non-empty before using it. Never ask the user to manually create API tokens — the CLI handles this automatically.

Investigation Workflow

Step 1: Parse Input

Extract organization, pipeline, and build number from the Buildkite URL or user input.

URL Pattern:

https://buildkite.com/{org}/{pipeline}/builds/{build_number}

The organization has multiple pipelines sharing the same agent pool:

• mockserver — primary Java build and test
• mockserver-client-node — Node.js client
• mockserver-node — Node.js module
• mockserver-performance-test — performance tests

Step 2: Get Build Overview

Using bk CLI (preferred):

bk build view {build_number} -p {pipeline} --json

Using REST API (fallback):

TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds/{build_number}"

Save the commit SHA and created_at — you will need these later to check whether fixes have already been pushed.

Step 3: Check Agent Availability

Using bk CLI:

bk agent list --json

This shows all agents across all pipelines, their connection state, and current job (if busy).

Key fields to check:

• connection_state — should be connected
• job — if present, agent is busy; check which pipeline/build it's running
• meta_data — agent tags including queue=default

Step 4: List Builds Across All Pipelines

To understand queue contention, check builds across all pipelines sharing the agent pool:

TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/builds?state[]=scheduled&state[]=running&per_page=50"

Step 5: Identify Failed Jobs

bk build view {build_number} -p {pipeline} --json

Filter for failed jobs in the JSON output. Look at jobs[].state == "failed".

Step 6: Retrieve Job Logs

TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds/{build_number}/jobs/{job_id}/log" \
  | jq -r '.content'

For large logs, save to file:

TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds/{build_number}/jobs/{job_id}/log" \
  | jq -r '.content' > .tmp/buildkite-{build_number}-log.txt

Step 7: Download and Analyze Build Artifacts

TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds/{build_number}/artifacts" \
  | jq '.[] | {filename,path,download_url}'

Download an artifact:

TOKEN=$(bk auth token)
curl -sLH "Authorization: Bearer $TOKEN" \
  "{download_url}" -o .tmp/buildkite-{build_number}-artifact.log

Step 8: Check GitHub Actions (Secondary CI)

If the Buildkite build passed but related GitHub Actions failed (CodeQL):

gh run list --repo mock-server/mockserver-monorepo --limit 10 --json status,conclusion,name,headBranch,createdAt
gh run view {run_id} --repo mock-server/mockserver-monorepo --log-failed

Step 9: Additional Investigation

Get build changes (commits that triggered the build):

git log --oneline {commit}..HEAD

Check recent build history for patterns:

TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds?per_page=20&state=failed" \
  | jq '.[] | {number,state,message,created_at}'

Cancel a build (e.g., stuck or blocking):

bk build cancel {build_number} -p {pipeline} -y

Rebuild a build:

bk build rebuild {build_number} -p {pipeline} -y

Step 10: Confirm Flaky-vs-Real

If the failure looks timing/ordering/port/resource-related (e.g. Timeout, Connection refused/BindException, intermittent assertion on order or timing), re-run the single failing test (or check recent builds of the same commit) to confirm intermittency BEFORE classifying it real-vs-flaky:

# Re-run only the failing build (does NOT change the commit under test)
bk build rebuild {build_number} -p {pipeline} -y

# Or check whether other builds of the SAME commit passed/failed
TOKEN=$(bk auth token)
curl -sH "Authorization: Bearer $TOKEN" \
  "https://api.buildkite.com/v2/organizations/mockserver/pipelines/{pipeline}/builds?commit={commit}&per_page=10" \
  | jq '.[] | {number,state,created_at}'

If it passes on re-run (or other builds of the same commit passed), classify it FLAKY and set reproduced: false. If it fails again deterministically, it is a real failure — set reproduced: true.

Step 11: Enumerate Competing Hypotheses

Before concluding a root cause, enumerate the competing hypotheses and the evidence that rules each out (correlation is not causation — a commit landing just before the failure is not proof it caused it). Record the survivors and the ruled-out alternatives in root_cause.alternative_hypotheses.

Step 12: Classify Error

Match the identified root cause against common failure patterns:

Step 13: Check for Already-Pushed Fixes

CRITICAL: Before recommending a fix, check whether the root cause has already been addressed:

git fetch origin master --quiet
git log --oneline {commit}..origin/master
git diff --name-only {commit}..origin/master

Classify:

Build Scheduling and Queue Behaviour

Understanding how Buildkite schedules builds is critical for diagnosing "stuck" builds:

• skip_queued_branch_builds — if enabled in the Buildkite pipeline settings (check via bk build view -p {pipeline} --json under pipeline.skip_queued_branch_builds), a newer build for the same branch causes older queued builds to be auto-skipped. This is normal, not a failure. The mockserver pipeline currently has this enabled.
• cancel_running_branch_builds — if disabled (check via pipeline.cancel_running_branch_builds), running builds are NOT automatically cancelled when a newer commit is pushed. Older running builds continue to consume agents. The mockserver pipeline currently has this disabled.
• Multiple pipelines share the same agent pool (queue=default). Scheduled builds from mockserver-performance-test, mockserver-client-node, and mockserver-node compete with the primary mockserver pipeline for agents.
• ASG max_size limits the total number of agents. The Lambda scaler cannot add instances beyond this limit. Check the current max with:

  DYLD_LIBRARY_PATH=/opt/homebrew/opt/expat/lib aws autoscaling describe-auto-scaling-groups \
    --auto-scaling-group-names "$(cd terraform/buildkite-agents && terraform output -raw auto_scaling_group_name 2>/dev/null || aws autoscaling describe-auto-scaling-groups --profile mockserver-build --region eu-west-2 --query 'AutoScalingGroups[?contains(Tags[?Key==`Name`].Value, `buildkite-mockserver`)].AutoScalingGroupName' --output text | head -1)" \
    --region eu-west-2 --profile mockserver-build \
    --query 'AutoScalingGroups[0].{MinSize:MinSize,MaxSize:MaxSize,DesiredCapacity:DesiredCapacity}'

Output — Structured Data Return

Return this structure in your final message:

{
  "schema": "pipeline-investigation/v1",
  "build": {
    "number": 0,
    "pipeline": "<pipeline slug>",
    "branch": "<branch>",
    "commit": "<commit sha>",
    "state": "failed",
    "failed_job": "<job name>",
    "started_at": "<ISO8601>",
    "finished_at": "<ISO8601>"
  },
  "reproduced": true,
  "root_cause": {
    "summary": "<one-line description>",
    "detail": "<technical explanation>",
    "error_excerpt": "<relevant log lines>",
    "failure_category": "<category from table above (e.g. FLAKY) or null>",
    "scope": "SYSTEMIC|ISOLATED|MAY_BE_SYSTEMIC",
    "alternative_hypotheses": [
      { "hypothesis": "<competing explanation>", "ruled_out_by": "<evidence that excludes it>" }
    ]
  },
  "fix_status": "OPEN|ALREADY_FIXED",
  "fix_commit": "<sha or null>",
  "fix_message": "<commit message or null>",
  "github_actions": [
    { "run_id": 0, "workflow": "<name>", "conclusion": "failure|success", "root_cause": "<summary or null>" }
  ],
  "commands_run": ["<exact bk / curl / git command used to gather evidence>"],
  "recommended_fix": "<actionable steps, null if already fixed>"
}

After returning the JSON, provide a brief summary (2-3 lines).

Notes

• Always investigate the deepest failure first (root cause in logs, not surface symptoms)
• Build artifacts often contain more detailed logs than the console output
• For recurring failures, check recent commits and pipeline definition changes
• Always run Step 13 before reporting — check if the issue was already fixed
• When builds are stuck/scheduled, check agent availability across ALL pipelines, not just the one being investigated
• Set reproduced: true only when the failure recurs deterministically (failed again on re-run, or every build of the commit failed); set reproduced: false for FLAKY failures that passed on re-run
• Always populate commands_run with the exact commands used so the conclusion can be replayed by the reader
• The bk CLI uses -p {pipeline} for pipeline selection, NOT --org — the org is set via bk auth switch