Debugging BuildStream Build Failures

Overview

Systematic guide for diagnosing and fixing BuildStream build failures in the bluefin-egg project. Covers all failure categories from source fetching through OCI assembly, plus CI-specific debugging.

Quick Reference

Diagnostic Flowchart

digraph debug {
    "Build failed" [shape=doublecircle];
    "Where in the pipeline?" [shape=diamond];

    "Source fetch/track" [shape=box];
    "Build commands" [shape=box];
    "Install commands" [shape=box];
    "Compose/filter" [shape=box];
    "OCI assembly (script)" [shape=box];
    "Cache issue" [shape=box];

    "Build failed" -> "Where in the pipeline?";
    "Where in the pipeline?" -> "Source fetch/track" [label="fetch error"];
    "Where in the pipeline?" -> "Build commands" [label="compile/configure error"];
    "Where in the pipeline?" -> "Install commands" [label="install error"];
    "Where in the pipeline?" -> "Compose/filter" [label="overlap/missing files"];
    "Where in the pipeline?" -> "OCI assembly (script)" [label="script element error"];
    "Where in the pipeline?" -> "Cache issue" [label="stale result / hash mismatch"];
}

Step 1: Read the error. BuildStream prints the failing element name and phase. Step 2: Run just bst artifact log <element> to get the full log. Step 3: Match to a category below.

Failure Categories

1. Source Fetch Failures

Symptoms: Failed to fetch, Connection refused, 404 Not Found, ref not found, Could not find base directory matching pattern

Debug command: just bst source fetch <element> -- isolates fetch from build.

2. Build Command Failures

Symptoms: configure failed, compilation error, command not found, missing header

Debug command: just bst shell --build <element> -- drops you into the sandbox with all build-deps staged. Run commands manually to isolate the failure.

3. Install Command Failures

Symptoms: No such file or directory, Permission denied, install: cannot create

Debug command: just bst shell --build <element> -- then manually run install commands to see exact error.

4. Compose/Filter Failures

Symptoms: overlap error, files missing from final image, FAILED: ... file already exists

Debug steps:

1. just bst show oci/layers/bluefin.bst -- check compose config, exclude lists
2. just bst artifact checkout <element> --directory /tmp/inspect -- inspect what the element actually installs
3. See oci-layer-composition skill for the full filter chain

5. Cache Corruption / Stale Artifacts

Symptoms: Build succeeds but produces wrong output, element shows as cached but content is outdated, hash mismatch errors

Key insight: BuildStream's cache keys include all dependencies. If you change a source ref but the element still shows cached, either the ref change didn't take effect or there's a junction caching issue. Run just bst show <element> and check Keys: -- if the key matches the old build, the source change wasn't picked up.

6. Junction / Upstream Failures

Symptoms: Element not found in junction, Failed to resolve, errors in freedesktop-sdk or gnome-build-meta elements

Debug: just bst show <junction>:<element> -- shows resolved element through the junction.

CI-Specific Debugging

Reading CI Logs

CI uploads two artifacts on every run (including failures):

Download from the GitHub Actions run page > Artifacts section.

on-error: continue

CI uses on-error: continue in buildstream-ci.conf. This means:

• BuildStream does not stop at the first failure
• It builds everything it can, skipping elements whose deps failed
• The build log contains ALL failures, not just the first
• Search the log for FAILURE to find every failing element

CI-Specific Failure Modes

Cache Proxy Debugging

The bazel-remote proxy bridges BuildStream gRPC CAS to Cloudflare R2:

• HTTP status: curl http://localhost:8080/status (from runner host)
• gRPC CAS: grpc://localhost:9092 (BuildStream connects here)
• Logs: /tmp/bazel-remote.log

If artifacts aren't being cached: check the proxy log for S3/R2 auth errors. The proxy config uses type: storage (not index) -- without this, BuildStream silently ignores the remote.

General Debugging Workflow

1. Identify the failing element -- BuildStream names it in the error output
2. Read the log -- just bst artifact log <element> (or download CI artifact)
3. Classify the failure -- match to a category above
4. Inspect the element -- just bst show <element> for deps, variables, commands
5. Reproduce interactively -- just bst shell --build <element> to run commands by hand
6. Fix and rebuild -- edit the .bst file, just bst build <element>
7. Clear cache if needed -- just bst artifact delete <element> before rebuilding

Related Skills

• local-e2e-testing -- running builds locally, prerequisites, environment variables
• buildstream-element-reference -- variable names, element kinds, source kinds
• oci-layer-composition -- how packages flow into the final image, compose filtering
• adding-a-package -- end-to-end workflow for new packages
• patching-upstream-junctions -- fixing upstream element issues via patches