signalbox integration evidence run

This is an evidence-gathering run, not a strict test suite. Every step runs best-effort: if a step fails or times out, record what happened (output + screenshot + verdict fail or warn) and move on. Never abort the run. The deliverable is a single self-contained HTML report with screenshots.

Expect the full run to take a long time - that is fine, it is designed as an overnight run.

Conventions (read first)

Paths and environment. All commands run from the repo root. One evidence dir per run:

EVIDENCE="$PWD/scratch/integration/run-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$EVIDENCE"
SB="$PWD/components/cli/bin/signalbox"

Every shell (shellwright sessions included) exports the test environment so the run never touches the real board:

export SIGNALBOX_DATA_DIR="$EVIDENCE/state"
export SIGNALBOX_CONFIG="$EVIDENCE/settings.json"
export SIGNALBOX_URL="http://127.0.0.1:8399"

SIGNALBOX_CONFIG matters as much as the data dir: without it the hub reads the user's real ~/.config/signalbox/settings.json, and a real hub.upstream/hub.token will leak into the test (the first hub start comes up as a forwarder to their remote). Pointing it at a non-existent file in the evidence dir means pure defaults.

Ports for this run: 8399 local hub, 8410 remote/upstream hub, 8420 forwarder. Space them out: LAN mode opens a second TLS listener on port+1 (8399 -> https on 8400, pinned self-signed cert), so adjacent ports collide. Token for authenticated scenarios: itest-token.

Evidence per step. Each step gets a directory $EVIDENCE/NN-slug/ containing:

• meta.json - {"title": "...", "verdict": "pass|warn|fail", "commands": ["..."], "notes": "one or two sentences: what was checked, what the evidence shows"}
• output.txt - captured command output (optional but preferred)
• *.png - screenshots, in the order they should appear

Screenshots of shell sessions come from shellwright's shell_screenshot; save or copy the PNG into the step dir. Screenshots of the app come from screencapture (see the app step).

Timing. Best effort. When waiting for output (hub startup, event propagation), poll shell_read every 1-2s up to ~15s rather than sleeping a fixed time. On timeout: screenshot anyway, verdict warn, continue.

Run metadata. At setup, write $EVIDENCE/run.json:

printf '{"started":"%s","commit":"%s","branch":"%s"}\n' \
  "$(date '+%Y-%m-%d %H:%M:%S')" "$(git rev-parse --short HEAD)" \
  "$(git rev-parse --abbrev-ref HEAD)" > "$EVIDENCE/run.json"

Step 01 - build

make build

Evidence: build output tail, $SB --version (or $SB help | head -3) showing the fresh binary. Also build the app bundle now so step 10 can use it (do not launch it yet):

make -C components/app app

Use the app target, NOT build: build only compiles Swift, while app rebuilds the bundle and embeds the fresh CLI at Contents/Resources/signalbox - a stale embed means the app-spawned hub reports the wrong version. App build failure is a warn on this step (the CLI scenarios still run) and turns step 10 into a fail.

Step 02 - local hub starts

Start a shellwright shell ("hub") with the test env exported, then:

$SB hub --port 8399

Poll until the startup banner appears. Screenshot the hub shell. In a second shellwright shell ("client", same env):

curl -s http://127.0.0.1:8399/healthz
$SB state

Evidence: healthz response, empty board, both screenshots.

Step 03 - seed the board

In the client shell:

SIGNALBOX="$SB" components/scripts/demo.sh
$SB state

Evidence: the seeded board showing every status colour (busy / attention / done / error). Screenshot $SB state output. Also capture $SB state --json head into output.txt as machine-readable evidence.

Then screenshot the jumplist showing the same seeded board, so the run has a picture of the surface people actually look at rather than only the CLI table. The app's hub is a different hub on 8377, so point the app at the test hub first, or seed the app's hub to match - note in meta.json which you did, because a jumplist of the real board is not evidence of this seed:

osascript -e 'tell application "System Events" to key code 38 using {control down, option down}'
sleep 1
screencapture -x "$EVIDENCE/03-seed/02-jumplist.png"

Do the same wherever a step changes what the board looks like - the session verbs in step 04 especially, where pin order, a rename and a hidden row are far clearer in the jumplist than in the CLI table. If the hotkey needs Accessibility permission and it is missing, record a warn and carry on.

Screen captures are full-screen and will include whatever else is open. Say so in the step note, and never paste the report anywhere shared without checking it.

Step 04 - session operations

In the client shell, exercise the session verbs against a seeded session key (e.g. claude:demo-schema-migration):

$SB session pin claude:demo-schema-migration
$SB session rename claude:demo-schema-migration "renamed by itest"
$SB session ack claude:demo-fix-auth-token-expiry
$SB session hide pi:demo-crash-analysis
$SB state
$SB session show pi:demo-crash-analysis
$SB state

Evidence: board screenshots showing the pin at top, the rename, the hidden row gone then back.

Step 05 - claude hook path

In the client shell, feed canned Claude Code hook payloads through the real adapter entry point (fields per components/cli/src/claude.ts: ClaudeHook):

echo '{"hook_event_name":"SessionStart","session_id":"itest-hook","cwd":"'"$PWD"'"}' | $SB hook claude
$SB state
echo '{"hook_event_name":"Stop","session_id":"itest-hook","cwd":"'"$PWD"'"}' | $SB hook claude
$SB state

Evidence: session appears busy after SessionStart, done after Stop. Hook commands must exit 0 even on failure - note the exit codes.

Step 06 - real agent via shellwright (optional, long)

Only if Claude Code hooks are installed for signalbox (check ~/.claude/settings.json for signalbox hook claude). Start a shellwright shell with the test env, run claude, send a trivial prompt ("say hi and stop"), wait for the turn to end, then check $SB state shows the session go busy then done. Screenshot the agent mid-turn and the board after.

If hooks are not installed, record the step as warn with a note and move on.

Step 07 - LAN mode (bind + token)

Stop the hub (Ctrl-C in the hub shell), restart it bound wide with a token:

SIGNALBOX_TOKEN=itest-token $SB hub --port 8399 --bind 0.0.0.0

This keeps loopback plain-http on 8399 and opens the TLS LAN listener on port+1 (https on 8400, pinned self-signed - hence curl -k). From the client shell, find the LAN IP (ipconfig getifaddr en0 or equivalent), then gather:

curl -sk -o /dev/null -w '%{http_code}\n' https://<lan-ip>:8400/state          # expect 401
curl -sk -o /dev/null -w '%{http_code}\n' -H 'Authorization: Bearer itest-token' https://<lan-ip>:8400/state   # expect 200
curl -s http://127.0.0.1:8399/healthz                   # loopback still fine, mode:lan

Then pairing: in the client shell run

SIGNALBOX_TOKEN=itest-token $SB pair

and screenshot the QR screen (Ctrl-C out of it after the screenshot).

Evidence: the 401/200 pair proves token enforcement; the QR screenshot proves pairing renders.

Step 08 - remote mode

In a third shellwright shell ("remote", same env). Give it its own data dir so its board is not shared with the local hub's:

SIGNALBOX_DATA_DIR="$EVIDENCE/state-remote" \
  SIGNALBOX_TOKEN=itest-token $SB hub --remote --port 8410

From the client shell:

curl -s http://127.0.0.1:8410/healthz                                   # open
curl -si http://127.0.0.1:8410/state | head -5                          # 401 - remote authenticates everything
SIGNALBOX_URL=http://127.0.0.1:8410 SIGNALBOX_TOKEN=itest-token \
  $SB fire --agent claude --event busy --session-key itest:remote --title remote-check
SIGNALBOX_URL=http://127.0.0.1:8410 SIGNALBOX_TOKEN=itest-token $SB state

Evidence: healthz open, state 401 without token, fired event visible with token.

Step 09 - forwarder and spool replay

Keep the remote hub (8410) running as upstream. Restart the hub shell as a forwarder with its own data dir (spool + read replica live there):

SIGNALBOX_DATA_DIR="$EVIDENCE/state-fwd" \
  SIGNALBOX_TOKEN=itest-token $SB hub --port 8420 --upstream http://127.0.0.1:8410

From the client shell:

SIGNALBOX_URL=http://127.0.0.1:8420 $SB fire --agent claude --event busy --session-key itest:fwd --title fwd-check
SIGNALBOX_URL=http://127.0.0.1:8420 $SB state                                        # read replica shows it
SIGNALBOX_URL=http://127.0.0.1:8410 SIGNALBOX_TOKEN=itest-token $SB state            # upstream shows it

Then the spool: Ctrl-C the remote hub (8410), fire again at the forwarder (must still succeed and exit 0 - it spools to state-fwd/forward-spool.jsonl; ls it as evidence). Note the spooled event does NOT appear on the read replica while the upstream is down - the replica mirrors upstream. Screenshot the forwarder shell log (drain retry errors are good evidence), restart the remote hub, and poll upstream state until the spooled event appears.

Evidence: event at both layers, fire-while-upstream-down succeeding, the spool file, replay arriving after restart. This is the core remote-hub story - screenshot generously.

Step 10 - the app

For rendering and screenshot checks, use a separate background app with the isolated environment above. Do not foreground a window or send global keys while the user works. The runtime override leaves the real app running, registers no shortcut or notification handler, and never saves the override.

open -g -n --env SIGNALBOX_URL="$SIGNALBOX_URL" \
  --env SIGNALBOX_DATA_DIR="$SIGNALBOX_DATA_DIR" \
  --env SIGNALBOX_CONFIG="$SIGNALBOX_CONFIG" \
  components/app/build/Signalbox.app --args \
  --show-jumplist --config-parameter ui.foreground=false

Use empty isolated transcript roots too, as in the CLI scenarios. Verify the test hub's /healthz reports the build just packaged. Locate the capture process's jumplist window ID through Core Graphics (filter by its PID and window bounds, not a global frontmost window), then capture only that window:

screencapture needs Screen Recording permission. If unavailable, record a warn; do not bring the app forward as a workaround.

screencapture -x -o -l "$CAPTURE_WINDOW_ID" "$EVIDENCE/10-app/jumplist.png"

Evidence: jumplist renders, trees start collapsed with descendant summaries, the original app/hub stays running, and foreground application and preferences are unchanged. Close only the capture process and its own hub afterwards. Background mode suppresses the menu icon and cannot test real keyboard focus. Record those checks as manual unless the user has allowed foreground interaction.

Step 10b - the iOS app (Simulator)

Prove phone-to-hub end to end: the iOS app in the Simulator connecting to the test hub and rendering the seeded board. Simulator loopback IS the Mac's loopback, so the app reaches the test hub with no pairing.

Note: xcrun simctl talks to the CoreSimulator service and typically needs the sandbox disabled on these Bash calls. No booted simulator / no iOS runtime installed = verdict warn with a note, move on.

# a booted device, or boot one
xcrun simctl list devices booted
xcrun simctl boot "iPhone 17 Pro"   # if none booted (any available iPhone works)

# build for the Simulator - do NOT pipe xcodebuild straight into tail/grep and
# move on: the pipe swallows a failure and you install a stale app. Check for
# BUILD SUCCEEDED explicitly.
cd components/ios && xcodebuild -project Signalbox.xcodeproj -scheme Signalbox \
  -destination 'platform=iOS Simulator,name=iPhone 17 Pro' \
  -derivedDataPath ../../scratch/ios-dd build

# the local hub (8399) was repurposed as a forwarder in step 09 - restart it
# in the hub shell with the test env; the seeded board replays from events.jsonl
$SB hub --port 8399

# install, point at the test hub, launch
APP=scratch/ios-dd/Build/Products/Debug-iphonesimulator/Signalbox.app
xcrun simctl install booted "$APP"
xcrun simctl spawn booted defaults write com.dwmkerr.signalbox.ios hubURL "http://127.0.0.1:8399"
xcrun simctl terminate booted com.dwmkerr.signalbox.ios 2>/dev/null
xcrun simctl launch booted com.dwmkerr.signalbox.ios
sleep 3
xcrun simctl io booted screenshot "$EVIDENCE/10b-ios/sessions.png"

After install, verify the sim runs the build you just made (the stale-install trap): md5 -q "$(xcrun simctl get_app_container booted com.dwmkerr.signalbox.ios)/Signalbox" "$APP/Signalbox" must match.

Evidence: the sessions page showing the green "Connected to 127.0.0.1" line and the seeded board rows. A screenshot right after launch (before the 3s sleep) additionally captures the quiet "Connecting..." state.

Step 10c - dropdown overflow

Seed at least 12 sessions against the app's own hub on port 8377. The demo seed provides six sessions; add distinct sessions so the overflow cap engages:

mkdir -p "$EVIDENCE/10c-dropdown-overflow" || true
SIGNALBOX_URL=http://127.0.0.1:8377 components/scripts/demo.sh || true
for session_number in 01 02 03 04 05 06 07; do
  SIGNALBOX_URL=http://127.0.0.1:8377 $SB fire --agent claude --event busy \
    --session-key "itest:overflow-$session_number" --title "overflow-$session_number" || true
done

osascript -e 'tell application "System Events" to tell process "Signalbox" to click menu bar item 1 of menu bar 1' || true
sleep 1
screencapture -x "$EVIDENCE/10c-dropdown-overflow/dropdown.png" || true

osascript -e 'tell application "System Events" to tell process "Signalbox" to perform action "AXShowMenu" of first menu item of menu 1 of first menu bar item of menu bar 1 whose name starts with "Show More"' || true
sleep 1
screencapture -x "$EVIDENCE/10c-dropdown-overflow/show-more.png" || true

If Accessibility or Screen Recording permission is missing, or the menu bar script cannot open the menu or submenu, record a warn with a note and move on. Evidence: the first screenshot shows exactly 10 inline rows in board order, followed by Show More (N) before the separator; the second shows the remainder in the opened submenu.

Step 10d - logs pane

Open Settings, select Logs, and capture the pane as soon as it opens. Use Accessibility scripting for the toolbar click; if it proves unreliable, select Logs manually and record that in the step note:

mkdir -p "$EVIDENCE/10d-logs-pane" || true
# Cmd-comma does NOT work here: a menu bar app has no window for the keystroke
# to reach, and System Events then errors with "Can't get window 1". Open
# Settings from the status menu instead.
osascript -e 'tell application "System Events" to tell process "Signalbox" to click menu bar item 1 of menu bar 1' || true
sleep 1
osascript -e 'tell application "System Events" to tell process "Signalbox" to click (first menu item of menu 1 of menu bar item 1 of menu bar 1 whose name starts with "Settings")' || true
sleep 2
osascript -e 'tell application "System Events" to tell process "Signalbox" to click button "Logs" of toolbar 1 of window 1' || true
sleep 1
screencapture -x "$EVIDENCE/10d-logs-pane/logs-current.png" || true

# Only toggle when the original appearance is known - toggling blind would
# leave the user's system flipped with nothing to restore to.
ORIGINAL_DARK_MODE="$(osascript -e 'tell application "System Events" to tell appearance preferences to get dark mode' 2>/dev/null || true)"
printf 'original dark mode: %s\n' "${ORIGINAL_DARK_MODE:-unknown}" > "$EVIDENCE/10d-logs-pane/appearance.txt"
case "$ORIGINAL_DARK_MODE" in
  true|false)
    osascript -e 'tell application "System Events" to tell appearance preferences to set dark mode to not dark mode' || true
    sleep 1
    screencapture -x "$EVIDENCE/10d-logs-pane/logs-opposite-appearance.png" || true
    osascript -e "tell application \"System Events\" to tell appearance preferences to set dark mode to $ORIGINAL_DARK_MODE" || true
    ;;
  *) printf 'appearance state unreadable; opposite-appearance capture skipped (warn)\n' >> "$EVIDENCE/10d-logs-pane/appearance.txt" ;;
esac

If Settings, appearance scripting, or screen capture fails, record a warn with a note and continue. Evidence: the Logs tail shows the newest lines, the timestamp column is visible, and the horizontal scroller is at the left, not mid-line. Both screenshots must be readable with no grey-on-white or black-on-black rendering. Restore the original system appearance.

Step 10e - shortcut recorder and migration

Capture Settings -> General with the recorder idle and the current shortcut's glyphs visible. Try the UI route first:

mkdir -p "$EVIDENCE/10e-shortcut-recorder" || true
# Snapshot the whole defaults domain before touching anything. The recorded
# shortcut is stored as Data, which per-key defaults read/write cannot
# round-trip - export/import round-trips every type.
defaults export com.dwmkerr.signalbox "$EVIDENCE/10e-shortcut-recorder/signalbox-defaults-backup.plist" || true

# Cmd-comma does NOT work here: a menu bar app has no window for the keystroke
# to reach, and System Events then errors with "Can't get window 1". Open
# Settings from the status menu instead.
osascript -e 'tell application "System Events" to tell process "Signalbox" to click menu bar item 1 of menu bar 1' || true
sleep 1
osascript -e 'tell application "System Events" to tell process "Signalbox" to click (first menu item of menu 1 of menu bar item 1 of menu bar 1 whose name starts with "Settings")' || true
sleep 2
osascript -e 'tell application "System Events" to tell process "Signalbox" to click button "General" of toolbar 1 of window 1' || true
sleep 1
screencapture -x "$EVIDENCE/10e-shortcut-recorder/recorder-idle.png" || true

Setting a combination through the recorder needs real keystrokes into the focused field; if that proves unreliable under osascript, record it as a manual step in the report rather than simulating it another way.

Test migration with the app quit. The domain snapshot above is the restore mechanism:

osascript -e 'tell application "Signalbox" to quit' || true
sleep 1
defaults delete com.dwmkerr.signalbox KeyboardShortcuts_openJumplist 2>/dev/null || true
defaults write com.dwmkerr.signalbox hotkey "cmd+shift+space" || true
open components/app/build/Signalbox.app || true
sleep 3
# The migrated value is Data holding JSON; defaults read prints a hex dump, so
# decode it back to text before asserting on the JSON keys.
defaults read com.dwmkerr.signalbox KeyboardShortcuts_openJumplist 2>/dev/null \
  | tr -d ' <>\n' | xxd -r -p \
  | tee "$EVIDENCE/10e-shortcut-recorder/migration-readback.txt" || true

osascript -e 'tell application "System Events" to tell process "Signalbox" to click menu bar item 1 of menu bar 1' || true
sleep 1
osascript -e 'tell application "System Events" to tell process "Signalbox" to click (first menu item of menu 1 of menu bar item 1 of menu bar 1 whose name starts with "Settings")' || true
sleep 2
osascript -e 'tell application "System Events" to tell process "Signalbox" to click button "General" of toolbar 1 of window 1' || true
sleep 1
screencapture -x "$EVIDENCE/10e-shortcut-recorder/migration-recorder.png" || true

# Restore the user's real preferences from the snapshot, whatever their types.
osascript -e 'tell application "Signalbox" to quit' || true
sleep 1
defaults import com.dwmkerr.signalbox "$EVIDENCE/10e-shortcut-recorder/signalbox-defaults-backup.plist" || true
open components/app/build/Signalbox.app || true

Evidence: the idle recorder screenshot shows the shortcut glyphs; the decoded migration readback contains carbonKeyCode 49 and carbonModifiers 768, which decode to ⌘⇧Space, and the migration screenshot shows the same. If the key, permissions, or any recorder operation fails, record a warn with a note and continue to the defaults import restoration. Note that the user's defaults domain was restored from the snapshot.

Close the Settings window with Cmd-W, never by clicking "button 1 of window 1". The toolbar buttons come first in the accessibility order, so button 1 is a pane button - on the Search pane that is "Rebuild Index...", which would kick off a full reindex of the user's real corpus.

Step 10f - lifecycle, search scroll, and removal regression

Run these palette regressions against a separate local hub and data directory. Do not point the test app at the real board. Use a spare port and give both the app and its embedded CLI the same isolated environment:

mkdir -p "$EVIDENCE/10f-palette-regressions" "$EVIDENCE/10f-state"
export SIGNALBOX_URL=http://127.0.0.1:8430
export SIGNALBOX_DATA_DIR="$EVIDENCE/10f-state"
export SIGNALBOX_CONFIG="$EVIDENCE/10f-settings.json"
SB="$PWD/components/cli/bin/signalbox"

# Background rendering leaves the real app and the user's focus alone.
open -g -n --env SIGNALBOX_URL="$SIGNALBOX_URL" \
  --env SIGNALBOX_DATA_DIR="$SIGNALBOX_DATA_DIR" \
  --env SIGNALBOX_CONFIG="$SIGNALBOX_CONFIG" \
  components/app/build/Signalbox.app --args \
  --show-jumplist --config-parameter ui.foreground=false

Wait for /healthz, then create three ordered rows, including a transcript whose opening lines precede the word scholar:

"$SB" fire --agent claude --event done --session-key itest:scholar-preview \
  --title "itest scholar preview" \
  --prompt $'first transcript line\nsecond transcript line\n\nscholar is later in this conversation'
for name in after target before; do
  "$SB" fire --agent claude --event done --session-key "itest:delete-$name" \
    --title "itest delete $name" --prompt "fixture for keyboard removal: $name"
done

The keyboard scenarios below require foreground interaction. When the user is working, record them as manual and keep the background screenshot checks; do not send global keys. For an explicitly allowed foreground run, relaunch the test app without the background override and record the real-app restart.

Open the jumplist for that app instance. Capture a search for scholar; the selected row should be at the top of the filtered list and the preview should show its opening lines, rather than opening at the transcript tail. Clear the search, press ⌃J once to select itest delete target, then press ⌃⌫; the adjacent row should remain selected and the removed row should disappear immediately. Capture both states.

For the disconnected marker, start a long-lived process, record its PID, fire an idle session event, terminate that process, then reopen the jumplist so its background process check runs:

sleep 600 &
DEAD_PID=$!
"$SB" fire --agent claude --event done --session-key itest:dead-process \
  --title "itest dead process" --pid "$DEAD_PID" --pid-name sleep
kill "$DEAD_PID"

Capture the row with the crossed-out terminal glyph. Also feed a Claude SessionStart followed by a non-clear SessionEnd and capture the separate fixture moved into the Hidden section:

printf '{"hook_event_name":"SessionStart","session_id":"itest-clean-end","cwd":"%s"}\n' "$PWD" | "$SB" hook claude
printf '{"hook_event_name":"SessionEnd","session_id":"itest-clean-end","reason":"other","cwd":"%s"}\n' "$PWD" | "$SB" hook claude

Save screenshots and meta.json under 10f-palette-regressions/; if Accessibility, Screen Recording, or the isolated app launch fails, record a warn with the exact limitation and keep the CLI regression tests as the machine-checked evidence.

Step 11 - teardown

Best effort: restore the original system appearance (10d) and the user's defaults domain via defaults import from the 10e snapshot if those steps changed them, then Ctrl-C / shell_stop every shellwright session, quit the app (osascript -e 'quit app "signalbox"' - adjust to the real app name), terminate the Simulator app (xcrun simctl terminate booted com.dwmkerr.signalbox.ios), and note anything left running. When killing a hub by pattern, use kill $(lsof -ti :PORT) - a pkill -f whose pattern appears in your own command line kills your own shell.

Step 12 - report

python3 .claude/skills/integration-test/build-report.py "$EVIDENCE"
open "$EVIDENCE/report.html"

The report is fully self-contained (screenshots base64-inlined) - safe to copy anywhere. Finish by telling the user the pass/warn/fail counts and the report path.