CI Pipeline Operations

Overview

The CI pipeline (.github/workflows/build-egg.yml) builds the Bluefin OCI image inside the bst2 container on Blacksmith runners, validates it with bootc container lint, and pushes to GHCR on main. Caching uses Blacksmith sticky disks (NVMe-backed Ceph) as primary storage, with GNOME upstream CAS (read-only, configured in project.conf) as a remote artifact source, and Cloudflare R2 as a read-only cold preseed for bootstrapping empty sticky disks.

Quick Reference

Workflow Steps

CI BuildStream Config

Generated as buildstream-ci.conf at step 8. Values and rationale:

Important: No remote artifact server is configured in buildstream-ci.conf. BuildStream uses only the local sticky disk cache and upstream GNOME caches defined in project.conf (read-only). Sticky disks auto-commit on job end -- no explicit upload step is needed.

Caching Architecture

Three layers, checked in order:

1. Sticky disk cache (~/.cache/buildstream/)
   Single NVMe-backed Ceph volume, persists across CI runs, ~3s mount
   Contains: CAS objects, artifact refs, source protos, source tarballs
   |-- miss -->
2. GNOME upstream CAS (https://gbm.gnome.org:11003)
   Read-only, configured in project.conf
   |-- miss -->
3. Build from source

Sticky Disk Details

Blacksmith sticky disks are NVMe-backed Ceph volumes that persist across CI runs. They auto-commit on job end (regardless of job outcome -- even failed builds persist their cache progress) and are evicted after 7 days of inactivity.

A single sticky disk holds the entire BuildStream cache:

Key behaviors:

• ~3 second mount -- negligible overhead vs. downloading a multi-GB cache archive
• Auto-commit on job end -- all changes written during the job are persisted automatically, even if the build fails
• 7-day eviction -- disks unused for 7 days are reclaimed; the R2 preseed step handles cold recovery
• Per-repository isolation -- disk keys include github.repository, so forks get separate disks
• Single volume inside container -- the Justfile mounts ~/.cache/buildstream at /root/.cache/buildstream inside the bst2 podman container; only this single volume is visible to BuildStream

R2 Cold Preseed

R2 is used only for cold cache recovery (when sticky disks are empty). The preseed step:

1. Checks if ~/.cache/buildstream/cas/ already has objects (warm disk = skip)
2. Installs rclone on-demand (not a separate workflow step)
3. Downloads cas.tar.zst from R2, validates with zstd -t, extracts
4. Syncs artifact refs from r2:bst-cache/artifacts/
5. Syncs source protos from r2:bst-cache/source_protos/

R2 is never written to during normal operation. The preseed data is a static snapshot. To refresh it, manually upload a new cas.tar.zst archive to R2.

Layer Summary

Trigger Modes

Schedule timing: The workflow runs daily at 08:00 UTC, after GNOME OS publishes artifacts (~03:03 UTC, done by ~04:00) and source tracking runs (06:00 UTC).

Secrets and Permissions

All R2 secrets are optional. If missing, the cold preseed step is skipped and the build proceeds using the sticky disk cache (if warm) plus GNOME upstream CAS. On a truly cold start without R2 secrets, everything builds from source -- slow but functional.

Job permissions: contents: read, packages: write.

bst2 Container Configuration

The bst2 container runs via podman run (NOT as a GitHub Actions container:), because sticky disk mounts must happen on the host before being bind-mounted into the container.

Note: --network=host is not needed since there is no local cache proxy. The bst2 container only needs network access for GNOME upstream CAS, which is accessed directly.

Debugging CI Failures

Where to Find Logs

Common Failures

Debugging Workflow

1. Check sticky disk mount: In the useblacksmith/stickydisk step output, verify the disk mounted successfully. If mounting fails, it's a Blacksmith infra issue.

2. Check preseed status: If the build is unexpectedly slow, check the "Preseed CAS from R2" step. If it ran, the sticky disk was cold. If it was skipped ("Sticky disk already warm"), the cache should have been populated.

3. Check disk space: Look at the "Cache and disk status" step -- it shows df -h and a breakdown of each cache component's size.

4. Search build log: Download buildstream-logs artifact and look for [FAILURE] lines in logs/build.log. on-error: continue means all failures are collected in one run.

5. Reproduce locally: just bst build oci/bluefin.bst uses the same bst2 container. See local-e2e-testing skill for full local workflow.

Cross-References