browser-matrix-strategy-reference

Overview

A real browser matrix tiers browsers by traffic share, regulatory requirement, and team budget, then runs each tier at its own cadence - not "test on everything".

This skill is a pure reference consumed by the sibling execution skills: cloud-grid-e2e (BrowserStack / Sauce Labs / LambdaTest), the self-hosted selenium-grid-4-runner, and playwright-testing browser projects (bundled engines; references/browser-matrix.md). Capping the matrix and publishing the support statement is covered in references/compatibility-budget.md.

When to use

• New project - defining the initial browser matrix.
• Quarterly review - re-tier browsers as traffic shifts (e.g., IE11 traffic drops to <0.1%, demote from T2 to T3).
• Audit / compliance - defending why specific browsers were tested (or weren't).
• Cost optimisation - moving low-tier browsers from cloud grid (paid) to bundled engines (free).

How to use

1. Pull browser / OS / version traffic share from your own analytics; supplement with StatCounter for geographies your own data is thin on.
2. Apply the tier-membership heuristic (defined once below) to place each combo in T1, T2, T3, or out of scope.
3. Group Chromium-engine browsers (Brave, Vivaldi, Opera) under the dominant browser instead of counting them as separate combos.
4. Map each tier to infrastructure via the cost-tier mapping: bundled engines for T1, cloud grid for T2 / T3, self-hosted Selenium Grid 4 for internal apps.
5. Record the matrix using references/matrix-template.md, giving every row an owner, a review date, and a "Why".
6. Log every tier change in the tier-change log so audit and "why is X here?" questions have a written answer.
7. Re-run quarterly: re-tier as traffic shifts, retire below-threshold combos, and defend legacy drops (IE11, old iOS Safari) against your own analytics.
8. Verify before dropping a combo: assert traffic <0.1% across two independent sources (own analytics + StatCounter) AND that no statutory or SLA requirement applies; if either check fails, keep the combo in T3 and re-review next quarter.

The three-tier model

Cadence per tier; exact traffic thresholds are defined once in the Tier-membership heuristic section below.

Data sources for traffic share

Use your own analytics for traffic-share decisions; supplement with StatCounter when slicing geographies you don't have own data for.

Worked example

A hypothetical SaaS B2B web app builds its first matrix from its own analytics:

1. Pull traffic share. Own analytics shows Chrome 68%, Firefox 12%, Safari 8%, Edge 5%, mobile Safari 9% of mobile traffic, mobile Chrome 5% of mobile, Safari iOS 16 2% of mobile and declining.
2. Apply the heuristic. Chrome 68% and Firefox 12% clear the T1 traffic bar, so both go to T1 (Chrome adds an N-1 row for Group-Policy lag-behind users). Safari 8%, Edge 5%, mobile Safari, and mobile Chrome fall in the T2 band or count as significant mobile platforms, so they go to T2. Safari iOS 16 at 2% and declining goes to T3.
3. Rule browsers out. A customer survey returns 0 IE11 users with no statutory requirement, so IE11 is out of scope; Brave and Vivaldi are Chromium-based and already covered by Chrome T1.
4. Map to infrastructure. T1 runs on bundled Playwright engines on CI (free, fast); T2 and T3 run on a cloud grid (real devices, nightly and quarterly).
5. Log the change. Edge grew 3 -> 5% since last quarter, so it is promoted from T3 to T2 and the promotion is recorded in the tier-change log.

Result: a 3-combo T1, 5-combo T2, 4-combo T3 matrix with a documented reason per row. The filled-in artifact is references/matrix-template.md.

Tier-membership heuristic

The single authoritative traffic thresholds, referenced by the three-tier model and worked example above. A browser belongs in:

• T1 if any of:

  • Currently ≥5% of own traffic, AND
  • "Most users" (statistically reasoned, not gut)
  • Browser the team uses internally (dogfooding)

• T2 if any of:

  • 1-5% of own traffic
  • Required by stakeholder commitment / SLA
  • Required by regulation (e.g., accessibility on Safari iOS for AppStore review)
  • Mobile platform with significant traffic

• T3 if any of:

  • <1% traffic but customer-requested
  • Long-tail enterprise (ESR / extended-support tracks)
  • Coverage-completeness for audit defensibility

• Out of scope if all of:

  • <0.1% traffic
  • No customer commitment
  • No regulatory requirement
  • Already covered by an engine-compatible browser (e.g., Brave covered by Chrome)

Cost-tier mapping

Where to test each tier:

For very high-volume + cost-sensitive teams, self-hosted Selenium Grid 4 for T1 + cloud grid for T2 / T3 is the optimum mix.

Engine vs runner distinction

A T1 entry "Chrome latest" can mean:

• Bundled Chromium engine in Playwright - fast, free, but engineered for testing (not 1:1 with stable Chrome)
• Real Chrome stable channel via cloud grid - slower, paid, but exactly what users have

For T1 the bundled-engine path is usually sufficient; for T2 / T3, real-browser via cloud grid is the choice when the difference matters (some bugs are real-Chrome-only).

Capping the matrix: the compatibility budget

The tiering above decides WHICH combos to test; the budget decides HOW MANY the team can afford to commit to, and publishes that commitment. references/compatibility-budget.md carries the four-tier commitment model (per-PR / nightly / pre-release / unsupported), the CI cost formula, example budgets per product type, the external "what we support" statement template, and the telemetry-to-tier mapping.

Matrix review checklist

When auditing a committed matrix (quarterly review, or before relying on it for a release), check four failure modes:

Refuse to treat a matrix as PASS while any BLOCK finding stands, and flag own-analytics data older than 90 days as stale telemetry.

Anti-patterns

Limitations

• Own-analytics blindness. Bot traffic + anonymised users can skew traffic counts; sanity-check via two sources.
• Cross-region traffic differs. Global average + your specific market may diverge; segment.
• Bundled engines aren't real browsers. Playwright's Chromium / WebKit have engineering-for-testing edits; for some bugs only real-browser cloud grids catch.
• Real-device matrices shift. Cloud grids retire devices / add new ones; quarterly check.

References

• StatCounter GlobalStats - gs.statcounter.com.
• MDN browser-compat-data - github.com/mdn/browser-compat-data.
• caniuse.com - caniuse.com.
• web-platform-tests / web-platform.org - web-platform-tests.org.
• Web feature status - webstatus.dev.
• GitHub Actions runner-images - github.com/actions/runner-images.
• W3C WebDriver specification - w3.org/TR/webdriver2/.
• Budget + support-statement reference - references/compatibility-budget.md and references/compatibility-budget-tiers.md.
• Sibling execution skills: playwright-testing (bundled engines; references/browser-matrix.md), selenium-grid-4-runner, cloud-grid-e2e (BrowserStack / Sauce Labs / LambdaTest).