smoke-suite-gate

Overview

A smoke suite is the minimum end-to-end test set every deploy must pass, gating pre-merge, post-merge to main, post-deploy to staging, and post-deploy to canary. A smoke failure halts the release.

When to use

• The CI/CD pipeline lacks a fast deploy gate; full regression takes too long for per-deploy.
• A canary stage needs a precursor smoke test that's faster than the full canary observation.
• A new release process needs the "first-line defense" check.

For broader coverage, see the team's full E2E suite (per qa-web-e2e plugin) - smoke is the narrow, fast subset.

Step 1 - Identify the critical paths

The smoke suite covers 5-15 journeys. Picking criteria:

1. High business value - broken = revenue / brand impact within minutes.
2. High traffic - most users hit this; broken = many users affected.
3. Cross-system - exercises multiple components; broken = integration regression.

Examples by product:

Step 2 - Implement fast

Smoke tests must run in <5 minutes total. Constraints:

// e2e/smoke/checkout.smoke.spec.ts
import { test, expect } from '@playwright/test';

test.describe('Smoke - checkout', () => {
  test('sign in → add to cart → checkout', async ({ page }) => {
    // 1. Sign in
    await page.goto('/login');
    await page.getByLabel('Email').fill(process.env.SMOKE_USER_EMAIL!);
    await page.getByLabel('Password').fill(process.env.SMOKE_USER_PASSWORD!);
    await page.getByRole('button', { name: /sign in/i }).click();
    await expect(page.getByRole('heading', { name: /welcome/i })).toBeVisible({ timeout: 10000 });

    // 2. Add to cart
    await page.goto('/products/SMOKE-001');
    await page.getByRole('button', { name: /add to cart/i }).click();
    await expect(page.getByTestId('cart-count')).toHaveText('1');

    // 3. Checkout
    await page.goto('/checkout');
    await page.getByLabel(/card/i).fill('4242 4242 4242 4242');
    await page.getByRole('button', { name: /place order/i }).click();
    await expect(page.getByRole('heading', { name: /order confirmed/i })).toBeVisible({ timeout: 15000 });
  });
});

Note: smoke tests use a pre-seeded test account, test-mode payment, and a known SKU (SMOKE-001). They don't create or delete data - pure read flows are best.

Step 3 - Pre-deploy vs post-deploy

Per stage, smoke acts as the "is this deploy worth proceeding with" gate.

Step 4 - Failure handling

A failing smoke isn't always a real regression - sometimes flake. Pattern:

- name: Run smoke
  id: smoke
  run: npx playwright test e2e/smoke/ --retries=2 --workers=2

- name: Quarantine repeat failure
  if: steps.smoke.outcome == 'failure'
  run: |
    if [ "${{ steps.smoke.conclusion }}" == "failure" ]; then
      # Real failure (failed twice with retries) - block deploy
      exit 1
    fi

The 2-retry rule kills most transients. A 3-retry-failure smoke test is either:

1. A real regression - block the deploy.
2. A genuinely flaky test - quarantine + investigate via flaky-test-quarantine (in the qa-flake-triage plugin).

Don't suppress failures by raising the retry count.

Step 5 - Smoke suite curation

The smoke suite must stay fast. Add tests deliberately:

Quarterly review: drop smoke tests that haven't caught a real regression in N quarters and aren't covering a new business value.

Step 6 - CI integration

# .github/workflows/smoke-gate.yml
name: smoke-gate
on:
  push:
    branches: [main]
  pull_request:

jobs:
  smoke:
    runs-on: ubuntu-latest
    timeout-minutes: 10   # hard cap - smoke must finish in 10 min
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v4
        with: { node-version: '20' }
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - name: Deploy ephemeral env (PR only)
        if: github.event_name == 'pull_request'
        run: ./scripts/deploy-ephemeral.sh ${{ github.head_ref }}
      - name: Run smoke
        env:
          SMOKE_USER_EMAIL: ${{ secrets.SMOKE_USER_EMAIL }}
          SMOKE_USER_PASSWORD: ${{ secrets.SMOKE_USER_PASSWORD }}
          BASE_URL: ${{ steps.deploy.outputs.url || 'https://staging.example.com' }}
        run: npx playwright test e2e/smoke/ --retries=2
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: smoke-results
          path: playwright-report/

timeout-minutes: 10 is the hard fail-fast - if smoke takes longer, something's wrong (the suite has bloated; deploy is slow).

Anti-patterns

Limitations

• Smoke ≠ regression. A green smoke doesn't prove the system works; it proves the critical paths work. Pair with full regression at lower frequency.
• Pre-seeded data dependencies. If SMOKE-001 SKU is deleted or the synthetic account is locked, smoke fails for the wrong reason. Monitor the test-data state.
• Test-mode payment vs real. Stripe / PayPal test mode behaves ~95% like prod; some prod-only edge cases need separate verification.
• Mobile smoke is different. Mobile uses mobile-device-matrix-toolkit (in the qa-mobile plugin) smoke tier; same principle, different platform.

References

• flaky-test-quarantine - handles repeat-failing smoke tests.
• prod-canary-validator - downstream gate after smoke passes.
• synthetic-monitor-author - same critical journeys, but continuous-in-prod (not per-deploy).