synthetic-monitor-author

Overview

Synthetic monitoring is "a monitoring technique that is done by using a simulation or scripted recordings of transactions" (synthetic-mon-wiki); the scripts "run continuously at set intervals to measure performance metrics like functionality, availability, and response time - without requiring actual traffic." Per the ISTQB Glossary V4.7.1, shift right is "a test approach to test a system continuously in production," and synthetic monitors are its load-bearing primitive. This skill builds the configuration: which journey, how often, what to assert, when to page.

When to use

• A critical user journey needs production-side coverage that doesn't depend on real user traffic (low-traffic SaaS, pre-launch, off-peak verification).
• A SLO depends on a specific user-facing flow being available; the monitor is the SLO-evidence source.
• An incident postmortem identified "we should have caught this in production faster" - the monitor is the prevention.
• A regulatory requirement (uptime SLA, healthcare availability) needs continuous active verification.

If real-user traffic is high and well-instrumented, real-user monitoring (RUM) is the complement.

Step 1 - Pick the journey

Synthetic monitors should target the highest-business-value journey the team would page on at 3am if it broke. Examples:

• E-commerce: search → add to cart → checkout → confirmation.
• SaaS: log in → access primary feature → save change.
• Financial: authenticate → fetch account balance → return.
• Healthcare: log in → view a patient record → log out.

Target commonly used paths and critical business processes. Don't monitor every flow - pick the 3-5 hero flows that map to the team's SLOs.

Which journeys to monitor - the RUM-coverage gap method

Synthetic monitors verify journeys the team chose to script; Real User Monitoring records journeys users actually take. The gap between the two sets is where production breakage goes undetected: a journey with 40 k sessions per day but no monitor can fail silently for hours. When RUM is instrumented, derive the journey list from it instead of gut feel:

1. Collect the RUM journey inventory. Pull the top ~50 view paths (or transaction names) by session volume from Datadog RUM, Sentry Performance, or GA4 + CrUX. Per-source queries are in references/rum-source-queries.md.

2. Score each journey: coverage_priority = session_volume_score x business_value_score, each on a 1-5 scale (range 1-25):

   Score	Daily sessions	Journey type (business value)
   5	> 10 k	Revenue-generating (checkout, upgrade); authentication (login, SSO, MFA)
   4	1 k - 10 k	Primary feature (core read/write); onboarding
   3	100 - 1 k	Support / self-service (docs, status)
   2	10 - 100	Informational (marketing pages, help)
   1	< 10	Admin / internal tooling

   The business-value column is editorial - align it with product stakeholders before the first run and record the agreed values.

3. Build the existing-monitor inventory. Datadog: GET /api/v1/synthetics/tests; Checkly: monitors/*.spec.ts + *.yml in the repo; New Relic: GET /v2/monitors.json. Normalize each monitor to a canonical URL path pattern (strip query strings, replace ID segments with {id}, lowercase).

4. Diff and rank. Journeys whose normalized path matches no monitor pattern form the gap list, sorted by coverage_priority descending. Emit one row per gap: path, sessions/day, business value, score, recommended monitor type (score >= 20 with interactions: browser check; pure API endpoint: API check; score < 10: defer - monitor sprawl costs more than the coverage is worth).

5. Feed the ranked gap list back into this step as the journey input.

Hard-reject rule: no RUM source, no gap analysis. If neither Datadog RUM, Sentry Performance, nor CrUX data exists for the target, halt and say so. Do not estimate journey volume from a sitemap - it contains every URL, not the ones users visit, and produces a monitor list biased by developer assumptions. Two data caveats: CrUX only captures publicly discoverable pages (use Datadog RUM or Sentry for post-login journeys), and Datadog RUM session retention is 30 days, so pick a representative date range.

Step 2 - Pick the platform

The platform decision typically follows the existing observability stack (Datadog APM → Datadog Synthetics; New Relic → New Relic Synthetics).

Step 3 - Author the script (browser check)

For browser checks, Playwright-style is the de-facto standard (Checkly natively, Datadog Synthetics increasingly). Drive the journey step by step with accessibility-first locators, then assert a confirmation state:

// monitors/checkout-journey.spec.ts (Checkly-style, excerpt)
import { test, expect } from '@playwright/test';

test('checkout journey - happy path', async ({ page }) => {
  await page.goto('https://example.com/');
  await page.getByRole('textbox', { name: 'Search' }).fill('BOOK-001');
  // ...search, add to cart, sign in with a synthetic account,
  //    place order with a test-mode card...
  await expect(page.getByRole('heading', { name: /Order confirmed/i })).toBeVisible();
});

Full browser and API templates: references/monitor-templates.md.

Use accessibility-first locators (not CSS classes); synthetic monitors that depend on CSS classes break on every UI refactor.

Critical: synthetic monitors hit production with real APIs. Use dedicated synthetic test accounts (not real customer data) and test-mode payment processors so the script doesn't trigger real charges / orders.

Step 4 - Author the script (API check)

For API checks, HTTP-step format chains requests and asserts on each step - status code, response shape, and response time:

# monitors/api-orders-flow.yml (Checkly-style, excerpt)
- name: 2. List orders
  method: GET
  url: https://api.example.com/orders
  headers: { Authorization: "Bearer {{TOKEN}}" }
  assertions:
    - { source: STATUS_CODE, comparison: EQUALS, target: 200 }
    - { source: RESPONSE_TIME, comparison: LESS_THAN, target: 500 }
    - { source: JSON_BODY, property: $.orders, comparison: IS_ARRAY }

Full multi-step auth + list + fetch template: references/monitor-templates.md.

Per-step assertions distinguish "the API returned" from "the API returned the right thing" - distinguish status code, response shape, and response time.

Step 5 - Cadence

Default: 5 min - matches most user journeys and fits within a 99.9% uptime SLO budget (5-min monitor with 2-failure alert rule gives ~10 min to detection, well within ~9 hours/year of allowed downtime). Use 1 min for the highest-criticality flows (auth, payment, primary read) or when the SLO is 99.99%+. Use 15 min for expensive E2E browser checks. Use 1 hour for transactions that have side effects. Use daily for compliance / audit verification flows.

Match the cadence to the SLO.

Step 6 - Alert thresholds

A single failure isn't an alert; a single failure is noise. Pattern:

• Page if N consecutive failures (typical N = 2 or 3).
• Page if M-of-K window (e.g., 3 of last 5 failed) - catches flapping monitors.
• Per-region: alert per geographic region; a single-region failure is often a CDN issue, not the application.
• Per-step: distinguish "the journey failed at step 1 (login)" from "the journey failed at step 4 (checkout)" - different on-call routing.

# Alert config (Checkly-style)
alerts:
  channels:
    - id: pagerduty-checkout
      filters:
        steps: [4, 5]   # only checkout/confirmation steps
    - id: slack-eng
      filters:
        consecutiveFailures: 1   # any failure → Slack notify
  escalation:
    runBased: true
    consecutiveFailures: 2
    cooldownPeriod: 1h

Step 7 - Locations

Run from multiple geographic regions (3-5 minimum):

• us-east, us-west, eu-west, ap-southeast, sa-east.

Response time varies dramatically by region; multi-region monitoring catches CDN / DNS / TLS issues that single-region misses.

Step 8 - As-code lifecycle

Treat monitors as code:

monitors/
├── checkout-journey.spec.ts       # browser check
├── api-orders-flow.yml             # API check
├── auth-flow.spec.ts
├── checkly.config.ts               # global config
└── README.md

CI pipeline (Checkly example):

- run: npm ci
- run: npx checkly test --reporter ci   # smoke check before deploy
- run: npx checkly deploy --force        # push the configs

Versioning the monitors in git means: PR review on changes, rollback if a monitor becomes flaky after a change, audit trail for why a monitor was added / removed.

Anti-patterns

Limitations

• Production load. Synthetic monitors generate traffic; at very small scale this matters (10 monitors × 1-min cadence = 14,400 requests/day per monitor).
• Doesn't cover real-user diversity. Synthetic monitors test what they're scripted to test; real users find what no one scripted. Pair with RUM.
• Maintenance burden. Monitors need updating when the product changes; broken monitors page on-call without product impact.
• Per-platform proprietary scripting. Datadog Synthetics scripts aren't Checkly scripts; lock-in is real. Prefer platforms that support Playwright-style scripts (more portable).

References

• synthetic-mon-wiki - Synthetic monitoring definition, active vs proactive vs real-user monitoring distinction, common metrics (Time to First Byte, Speed Index, Time to Interactive, Page Complete), named providers (Datadog, F5).
• references/rum-source-queries.md - per-source queries for the RUM-coverage gap method (Datadog RUM Explorer, Sentry Performance throughput, GA4 + CrUX field data), with the Datadog retention and CrUX public-discoverability caveats cited at the point of use.
• ISTQB Glossary V4.7.1 - https://glossary.istqb.org/en_US/term/shift-right defines shift right as "A test approach to test a system continuously in production." (Per workspace memory: ISTQB glossary is JS-rendered; navigate via Playwright or real browser.)
• feature-flag-experiment-validator - sibling skill: validates A/B experiments running behind flags.
• prod-canary-validator - sibling: catches regressions in canary stage before full rollout.