OS / runtime matrix

Reference for github-actions-test-jobs: run tests across operating systems (Linux / macOS / Windows) and runtime versions (Node 18/20/22; Python 3.10/3.11/3.12; Java 17/21; .NET 6/8) with GitHub Actions matrix syntax, and address the OS-specific quirks (path separators, line endings, file permissions, shells) that break cross-platform suites.

When to use

• The product runs on multiple OSes (CLI tools, libraries, desktop apps).
• The team supports multiple runtime versions.
• A bug report says "works on Linux, broken on Windows."

For browsers specifically, see the qa-web-e2e plugin's playwright-testing (references/browser-matrix.md).

Step 1 - Define the OS matrix

# .github/workflows/os-matrix.yml
strategy:
  fail-fast: false
  matrix:
    os: [ubuntu-latest, macos-latest, windows-latest]

GitHub Actions provides:

Step 2 - Define the runtime matrix

Per language:

# Node.js
strategy:
  matrix:
    node: [18, 20, 22]
    os: [ubuntu-latest, macos-latest, windows-latest]

# Python
strategy:
  matrix:
    python: ['3.10', '3.11', '3.12']
    os: [ubuntu-latest, macos-latest, windows-latest]

# Java
strategy:
  matrix:
    java: [17, 21]
    os: [ubuntu-latest, macos-latest, windows-latest]

# .NET
strategy:
  matrix:
    dotnet: ['6.0.x', '8.0.x']
    os: [ubuntu-latest, macos-latest, windows-latest]

The full cross-product is 3 OSes × 3 runtimes = 9 jobs. For larger matrices, use include + exclude to skip uninteresting combinations.

Step 3 - Address OS-specific quirks

Path separators

// Bad - hardcoded /
const configPath = projectRoot + '/config/app.json';

// Good - path.join
const path = require('node:path');
const configPath = path.join(projectRoot, 'config', 'app.json');

Line endings

# .gitattributes
*.sh text eol=lf
*.bat text eol=crlf
*.json text

Without .gitattributes, Windows users may commit CRLF; tests that compare output strings break.

Case sensitivity

// On Linux: import './Utils' fails if file is './utils'
// On macOS / Windows (default): both work

// Always match file case exactly:
import { foo } from './utils';   // matches utils.js

Shell

- name: Run script (cross-platform)
  shell: bash
  run: ./scripts/setup.sh

shell: bash works on Linux + macOS + Windows (via Git Bash on Windows runners).

Step 4 - Per-OS conditional steps

When OS-specific setup is needed:

- name: Install Linux deps
  if: runner.os == 'Linux'
  run: sudo apt-get install -y libssl-dev

- name: Install macOS deps
  if: runner.os == 'macOS'
  run: brew install openssl

- name: Install Windows deps
  if: runner.os == 'Windows'
  run: choco install openssl

Step 5 - Aggregate per-OS results

## OS / runtime matrix results - `<sha>`

| OS        | Runtime  | Tests | Pass | Fail | Time |
|-----------|----------|------:|-----:|-----:|-----:|
| Linux     | Node 22  |  142  |  142 |    0 | 2m   |
| Linux     | Node 20  |  142  |  142 |    0 | 2m   |
| Linux     | Node 18  |  142  |  140 |    2 | 2m   |  ← Node 18 incompat
| macOS     | Node 22  |  142  |  141 |    1 | 3m   |  ← macOS path issue
| macOS     | Node 20  |  142  |  141 |    1 | 3m   |
| Windows   | Node 22  |  142  |  140 |    2 | 4m   |  ← Windows path issue
| ...

Step 6 - Per-OS conditional tests

Some tests are OS-specific:

// jest.config.js
module.exports = {
  testPathIgnorePatterns: process.platform === 'win32'
    ? ['unix-only.test.js']
    : ['windows-only.test.js'],
};

Or via test framework conditionals:

test.skipIf(process.platform === 'win32')('uses fork()', () => {
  // POSIX-specific test
});

Step 7 - Cost management

Matrix size grows multiplicatively. Manage cost:

The "smoke matrix" per-PR keeps CI cheap; the full matrix runs less frequently.

Anti-patterns

Limitations

• GitHub Actions runner cost. Windows + macOS runners are more expensive than Linux; matrix design tradeoff.
• Per-OS bugs may surface only at runtime. Static analysis catches some; integration tests are the safety net.
• macOS-specific issues often only reproduce on real macOS; Linux CI doesn't catch them.
• Older OS versions rarely available on hosted runners; need self-hosted for legacy.

References

• GitHub Actions runners docs at docs.github.com/en/actions/using-github-hosted-runners.
• playwright-testing (in the qa-web-e2e plugin, references/browser-matrix.md) - sibling: browser-specific.
• browser-matrix-strategy-reference (in the qa-web-e2e plugin, references/compatibility-budget.md) - conventions for matrix sizing.