ci-test-job-conventions

Overview

Per-CI platform skills (GitHub Actions / GitLab CI / Jenkins / CircleCI) cover how to express workflows. This skill covers what to express - the cross-platform conventions that apply regardless of CI tool.

When to use

• Designing the CI test workflow for a new repo or service.
• Auditing an existing CI config for anti-patterns - retries that mask flake, no concurrency-cancel, artifacts retained forever.
• PR review of a workflow / .gitlab-ci.yml / Jenkinsfile change.
• Standardizing test-job conventions across teams on different CI tools.

How to use this reference

1. Size the suite, pick the shard count from the sharding matrix (§1) and set the per-job timeout for each job type (§8).
2. Set the retry policy - which failure classes are safe to retry vs. quarantine as flake (§2, §3).
3. Add concurrency-cancel so a new push cancels the superseded run (§6).
4. Tier the per-trigger cadence - what runs per-PR vs per-merge vs nightly vs pre-release (§5).
5. Emit JUnit XML and set artifact retention - the reporting and cache lookup tables live in references/junit-and-cache-lookups.md; retention bands in §4.
6. Guard secrets with the CI secret store, never in-repo (§7).

Then re-check the workflow against the conventions below.

§1 - When to shard

Sharding splits a test suite across N parallel jobs. Decision matrix:

Sharding cost-equivalent is N parallel × ~runtime/N - same total CPU-time, faster wall-clock.

§2 - Retry policy

Distinguish retry classes:

Rule: Maximum 1 framework-level retry. More retries hide flake.

§3 - Flake quarantine integration

Failed-on-first-run-passed-on-retry tests are flake. Pattern:

1. Test fails → CI marks @flaky tag.
2. Continued failures over N runs → quarantine (move to separate suite that doesn't gate).
3. Periodic review of quarantined tests → fix or delete.

Per flaky-test-quarantine (in the qa-flake-triage plugin) for the workflow.

§4 - Artifact lifecycle

# Recommended retention per artifact type
test-results:        14 days   # short-term debugging
coverage-reports:    30 days   # trend analysis
e2e-screenshots:     30 days   # failure debugging
performance-traces:  90 days   # historical analysis
deployment-logs:     90 days   # audit / compliance

Per-CI:

Don't retain forever - storage cost.

§5 - Per-trigger filtering

Per-PR (push to PR branch):
  - Smoke tests.
  - Lint + unit tests.
  - Per-changed-files coverage gate.

Per-merge to main:
  - Full unit + integration suite.
  - Smoke E2E.
  - Coverage trend tracking.

Per-deploy to staging:
  - Smoke E2E against staging.
  - Synthetic monitor smoke.

Nightly scheduled:
  - Full E2E across browsers.
  - Full security scans (axe, OWASP).
  - Mutation testing (per `stryker-mutation`).

Pre-release tag:
  - Cross-platform matrix (per `mobile-device-matrix-toolkit`).
  - Cross-browser matrix (per `browser-matrix-runner`).
  - Manual UAT sign-off.

Manual / on-demand:
  - Specific debug runs.
  - Performance / load tests.

Tier the cadence to balance feedback latency vs cost.

§6 - Concurrency control

When PRs receive rapid pushes:

The pattern: cancel superseded runs. Saves CI cost on stale commits.

§7 - Secret management

Never:
- Commit credentials to .yml / Jenkinsfile
- Use secrets in pull_request from forks
- Use `set -x` in scripts that handle secrets

Always:
- CI platform's secret store
- Mask in logs (`echo "::add-mask::$VALUE"` for GHA)
- Rotate on schedule
- Scope per-job (job-level env > workflow-level env > global)

§8 - Per-job timeouts

Hard timeouts prevent runaway jobs from consuming runners.

§9 - Cross-CI portability

If the team needs CI portability (multiple CIs in use, or anticipates migration):

• Encapsulate logic in shell scripts (scripts/test.sh, scripts/build.sh); CI calls the script.
• Use standard env vars (CI=true, CI_BRANCH, CI_COMMIT_SHA); abstract per-CI vars.
• Containerize the build (Docker image with all deps); CI just runs the container.

The goal: .github/workflows/test.yml, .gitlab-ci.yml, and Jenkinsfile are thin wrappers calling the same scripts.

Worked example: a 25-minute Playwright E2E suite on GitHub Actions

Walk the How-to-use steps for a suite that runs 25 minutes end-to-end.

1. Shard + timeout. 25 min falls in the 10-30 min band (§1), so split into 4 shards - roughly 6-7 min wall-clock each - and set a 30 min hard timeout per shard job (the E2E per-browser row in §8).
2. Retry. Allow at most 1 framework-level retry for a network timeout to a dependency (§2); a test that only passes on that retry is flake, not a pass - tag it @flaky and quarantine it (§3) rather than retrying further.
3. Concurrency. Set concurrency: group: ${{ github.workflow }}-${{ github.head_ref }} with cancel-in-progress: true (§6) so a fresh push to the PR cancels the superseded run.
4. Cadence. Run a smoke subset per-PR, the full 4-shard E2E per-merge to main, and the full cross-browser E2E nightly (§5).
5. Reporting + artifacts. Emit JUnit XML via dorny/test-reporter so results feed junit-xml-analysis (references/junit-and-cache-lookups.md), and upload failure screenshots with retention-days: 30 - the e2e-screenshots band in §4.

Deep references

The per-CI reporting mechanism and the per-language reporter and dependency-cache lookups live in one companion reference so this file stays a decision surface:

• Reporting + cache lookups - JUnit XML support per CI, default reporters per language, and per-language cache-key recommendations: references/junit-and-cache-lookups.md.

References

• github-actions-test-jobs, gitlab-ci-test-jobs, jenkinsfile-test-stages, circleci-test-configs - per-CI implementation skills.
• flaky-test-quarantine - flake handling.
• junit-xml-analysis - JUnit XML parser.
• e2e-suite-budget - when to refactor instead of shard more.
• Reporting + cache lookups (per-CI JUnit, per-language reporters + cache keys): references/junit-and-cache-lookups.md.