Adding Tests and CI

Why this is a cost question

Every agent-native workflow draws on one org-wide pool of GitHub-hosted runners. On the Team plan it held 60 concurrent jobs, and on 2026-09-29 the Fast tests gate waited a median 445 s just for a slot. BuilderIO moved to Enterprise Cloud in October 2026, which raises the pool to 500 jobs (50 macOS). Minutes are free for this OSS repo; slots are shared with every private repo. Scope still matters, because a job a change cannot affect delays other PRs once the pool fills, but parallelism that cuts the critical path is now worth a slot.

scripts/ci-change-scope.ts is the one place that decides what a change runs. It classifies the changed paths into a full or targeted run and emits one output per check. ci.yml jobs gate on those outputs. A .github/** change, the lockfile, root config, or a scripts/** change other than a guard script or root script test forces a full run.

Adding a test

First decide whether the test should exist: test-audit owns that gate and its junk patterns. This section covers where it lives and what runs it.

Put it in the workspace of the code it proves. Targeted runs test the changed packages and their dependents (...{packages/core}), so a test in another package either runs on unrelated changes or never runs on the one that breaks it.
Prove it at the cheapest tier that can fail for the bug: a pure function, then a component or .db.test.ts, then browser E2E last. Some E2E suites, such as Design's, run only after merge and cannot fail a PR. Do not restate one assertion at several tiers. design-editor-architecture covers failing-first proof and mutation audits.
A new test joins an existing lane or job. It never gets a job of its own. A changed root scripts/*.test.ts, or the test beside a changed guard script, runs in Security guards through the script_tests output.
Mind what vitest --changed cannot see. Targeted Core lanes run only the tests whose import graph reaches a changed file. A test that reads a file from disk (a skill, a fixture, a template) will not rerun when only that file changes. Import it, or teach requiresFullCoreFastTests in scripts/ci-test-lanes.ts about the path.
Never skip, disable, or quarantine a test to get green.

Writing a browser E2E test

A browser test is the most expensive tier, so it has to fail when the feature breaks the way a user would see it. A test that only checks that something happened stays green while the result is wrong.

Assert the exact end state, stated literally: the saved parent, order, position, and size. A fired message, toContain, a file over N bytes, or a loose closeTo passes over the bug.
Read persisted state, not only the DOM. Wait for the save to settle, then read what the server stored. An edit that never saved must fail.
Fail on collateral change. Diff the saved document before and after, and allow only the nodes the step meant to touch.
Fail on app errors by default, in a fixture: console and page errors, app responses of 400 or more, error toasts, a save still in flight at teardown, and page loads the test did not start. A request that fails and is then retried still counts.
Own the data. Create a fresh record per test and delete it after the page leaves. Never mutate a shared seed or delete a record under an open editor.
Drive the real input path at any zoom. Click, drag, and type at positions computed from the app's own geometry; constant page-pixel offsets break when the default zoom changes.
One flow per test, a guard before each step, no conditionals, no sleeps. In a serial chain, one failure skips everything after it.
Keep artifacts out of the watched source tree. A file written there reloads the dev server mid-test; use testInfo.outputPath.
Check intent before calling it a bug. Read the source first; some saved attributes that look like leaks are deliberate metadata. A test that asserts correct behaviour and fails is a bug report: never loosen it to pass.

Design's harness (templates/design/e2e/rails.ts) and its traps are in references/design-e2e.md.

Test workers

Fast-test lanes run Vitest with VITEST_CONCURRENCY=100%, one worker per core. The shared config's 25% default is for laptops, and on CI's 4-vCPU runners it meant one worker: a core shard took 542 s at one worker and 201 s at four, on the same CPU time. On the 60-slot pool that let two full-width lanes replace five, but each lane then ran ~18 min of packages one after another. With 500 slots CI plans eight lanes (LANES in ci.yml): each costs ~100 s of setup, and the planner balances lanes by test-file count.

Large packages are sharded too. splitLargePackages in scripts/ci-test-lanes.ts splits a package heavier than a fair lane share into Vitest --shards, so Design no longer sets the floor for every lane. Only a bare vitest test script can be sharded, and no shard drops below MIN_SHARD_FILES.
Let CI override the worker count. A package that pins maxWorkers goes through resolveMaxWorkers(process.env, fallback); a literal overrides the CI setting.
Leave headroom for work inside a test body. Booting PGlite or importing a server bundle takes several times longer when every core is busy, and a different test crosses the limit each run. The shared config allows 30 s; do not add a stricter per-block timeout.
Give parallel workers separate databases. A file that opens the database without DATABASE_URL gets the default directory, whose lock admits one process at a time. Core's Vitest setup gives each worker slot its own under a root that belongs to one run, so concurrent runs never share one; a test that asserts the default URL clears DATABASE_URL itself.
Keep the forks pool with isolation. Threads ran slower and failed 79 tests; turning isolation off gained 7% and failed 129.

Adding or changing a CI job

Rule	Why, from this repo
Give a job its own check in `CHECK_NAMES` whose condition names the paths that can change its result. Never borrow another job's output.	The Postgres connection budget reused `neon_query_budget` and ran on every template edit. Its probe imports only `@agent-native/core/db`.
Per-app work takes a list output, not a boolean. Shared packages or a full run select every app; a template change selects that template; an empty selection turns the check off. If the check is on and the list is empty, the job fails.	`query_budget_apps` and `ssr_boot_apps`: a one-template PR built and measured all 16 templates (12–13 min). When shared code selects every template, `query_budget_matrix` splits them across two jobs. The beta publisher's `discover-sites` publishes only sites whose dependency closure changed.
Split the expensive step from the cheap one. Keep the cheap check on every run and gate the expensive step on its narrower inputs.	Android: `expo export` runs whenever a file Metro bundles changes, but the Gradle compile, the bulk of the ~23 min Android job, runs only when `packages/mobile-app`, the lockfile, or the workflow changed.
A path filter covers the job's whole dependency closure, including install-time inputs: root `package.json`, `pnpm-workspace.yaml`, the prebuild script, and the lockfile.	The desktop canary filter missed the bundled Chrome extension and then the postinstall inputs. Each gap skipped runs that should have caught a break. `guard:mobile-build-paths` traces Metro's imports and fails when the mobile filter misses one.
Subscribe only to events that can change the outcome. Validators pin some trigger sets, e.g. `scripts/validate-*-workflow.ts` and `scripts/package-release-workflow.test.ts`; update them in the same change.	Content product conformance reran on `labeled`/`unlabeled` but never reads labels, and on title edits although it reads the body only for its declaration.
Do not add a job for under a minute of work. Checkout, install, and a pool slot cost more than the check. Fold it into a job with the same setup, and do not duplicate what `pnpm guards` already runs.	Consolidation took PR pushes from ~31 to ~23 jobs. It folded PGlite locking into Content DB tests, privacy evals into Brain evals, QA static into Security guards, and the changeset check into Lint & format, and dropped a drizzle guard job `pnpm guards` covered.
Batch periodic publishing on a schedule with change detection, instead of once per merge.	Nightly npm snapshots publish every 3 h, and only when a publishable path changed since the last successful scheduled run.
PR workflows cancel superseded runs: `group: <name>-${{ github.event.pull_request.number \|\| github.ref }}` with `cancel-in-progress: true`. Publishers on `main` use `cancel-in-progress: false` so a deploy is never killed mid-flight. An event rejected only by a job `if` still joins the run's group and cancels a real run, so filter it in `on:` or give ignored events a throwaway group.	Visual Recap events the gate ignored used to cancel an in-progress recap; they now get a throwaway group.
When scope cannot be computed, run everything or fail. Never skip.	An unreadable beta site list, an unknown base, or unrelated history publishes every site. A failed `git diff` fails the step; it is never `\|\| true`.
Print what fails, not everything.	The full-tree lint printed ~15k pre-existing warnings over the one error. It now runs `oxlint --quiet`.

A skipped job counts as passing for required checks, so gating a required job or step with if is safe. Every uppercase key a workflow adds under env: needs a docs/environment-variables.md row (guard:env-documentation). Prefer built-in GITHUB_* variables over new ones.

Proving a CI change

Classifier: add scripts/ci-change-scope.test.ts cases for each new check: on, off, full run, and tooling-only full run. Then run node --experimental-strip-types --test scripts/ci-change-scope.test.ts scripts/ci-build-workspaces.test.ts scripts/ci-test-lanes.test.ts.
Real history: run scripts/ci-change-scope.ts with GITHUB_OUTPUT set on real main commit ranges, and put the before/after in the PR.
Shell: run each new run: snippet locally against a matching input and an empty one. Run actionlint on the workflow when it is available.
Mind the self-test gap: a PR that edits ci.yml runs full CI itself, so its own checks cannot show the saving. The classifier output is the evidence.

Real failures this replaces

"whats up with Cold-request query budget. runs on every CI for every template? seems wrong"
"fast lane tests seem absuredly expensive"
"why is Generate + run standalone Chat running so frequently? shouldnt it listen to Chat changes?"
"so tldr: ~2-3 simultaneous PRs will eat up our entire quota of jobs right"
"can you make lint & format not print all the warnings?"