rum-to-synthetic-gap-analyzer

Overview

Synthetic monitors verify journeys that the team chose to script. Real User Monitoring records journeys that users actually take. The gap between the two sets is where production breakage goes undetected: a journey with 40 k sessions per day but no synthetic monitor can fail silently for hours before on-call is paged.

This skill closes that gap. It reads RUM journey data, scores each journey by traffic volume times business value, diffs the result against the team's existing synthetic monitor inventory, and emits a ranked gap list that synthetic-monitor-author can consume directly.

How to use

1. Pull the top ~50 journeys by session volume from your RUM source - Datadog RUM, Sentry Performance, or GA4 + CrUX (Step 1).
2. Score each journey coverage_priority = session_volume_score x business_value_score on the 1-5 tables (Step 2).
3. Build the inventory of active synthetic monitors and normalize each to a canonical URL path pattern (Step 3).
4. Diff the scored journeys against the monitor inventory to isolate the uncovered paths (Step 4).
5. Sort the uncovered paths by coverage_priority and emit the Markdown gap report, one row per gap (Step 5).
6. Add a recommended monitor type per row using the score heuristic, then hand the ranked list to synthetic-monitor-author.

Step 1 - Collect the RUM journey inventory

Pull the top-N view paths (or transaction names) by session volume from the active RUM source. Aim for the top 50 to avoid chasing long-tail pages that carry negligible traffic.

Per-source query instructions - Datadog RUM Explorer grouping, Sentry throughput sorting, and GA4 + CrUX field data (LCP/INP/CLS at p75) - are in references/rum-source-queries.md.

Step 2 - Score each journey

Assign each journey a coverage-priority score:

coverage_priority = session_volume_score x business_value_score

Session volume score (1-5): rank by daily session or view count.

Business value score (1-5): assign by journey type. Adjust to your domain.

Score range: 1 (low-traffic, low-value) to 25 (highest-traffic, revenue-critical).

For public-facing sites, supplement volume score with CrUX visit share where available: higher CrUX weight indicates broader real-user exposure.

Step 3 - Build the existing-monitor inventory

Collect the names or URL patterns of every active synthetic monitor. Most platforms expose this via API or config file:

• Datadog Synthetics: GET /api/v1/synthetics/tests returns all tests with their config.request.url and type (browser or api).
• Checkly: checkly tests list --output json or read monitors/*.spec.ts and monitors/*.yml in the repo.
• New Relic: GET /v2/monitors.json via NR REST API.
• Checkly as-code layout (per synthetic-monitor-author): monitors/ directory contains .spec.ts (browser) and .yml (API) files.

Normalize each monitor to a canonical URL path pattern (strip query strings, replace ID segments with {id}, lowercase). Store as a set.

Step 4 - Diff: rank the gap list

For each journey in the scored list (Step 2), check whether the normalized path matches any pattern in the monitor inventory (Step 3).

gap_list = [j for j in scored_journeys if not matches_any_monitor(j.path)]

Sort gap_list descending by coverage_priority. The output is the gap list.

Step 5 - Emit the gap report

Output format (Markdown table, one row per gap):

| Rank | Journey path       | Sessions/day | Biz value | Score | Recommended monitor type |
|------|--------------------|--------------|-----------|-------|--------------------------|
| 1    | /dashboard         | 12 k         | 4         | 20    | Browser (multi-step)     |
| 2    | /onboarding/step1  | 5 k          | 4         | 16    | Browser (multi-step)     |
| 3    | /reports/{id}      | 1.2 k        | 4         | 12    | Browser (read + assert)  |

Recommended monitor type heuristic:

• Score >= 20 and has user interactions: browser check (Playwright-style).
• Score >= 20 and is a pure API endpoint: API check (HTTP steps).
• Score 10-19: browser check if it has a UI; API check otherwise.
• Score < 10: flag for deferred coverage; do not generate a monitor yet.

Pass the gap list to synthetic-monitor-author as the journey input for Step 1 of that skill.

Worked example

Scored journeys (top 5):
  /checkout         vol=5, biz=5  -> score 25  [MONITOR EXISTS: checkout-journey.spec.ts]
  /dashboard        vol=5, biz=4  -> score 20  [NO MONITOR]  <-- gap rank 1
  /login            vol=4, biz=5  -> score 20  [MONITOR EXISTS: auth-flow.spec.ts]
  /onboarding/step1 vol=4, biz=4  -> score 16  [NO MONITOR]  <-- gap rank 2
  /reports/{id}     vol=3, biz=4  -> score 12  [NO MONITOR]  <-- gap rank 3

Gap list (ready for synthetic-monitor-author):
  1. /dashboard         score=20  sessions/day=12k  business=primary feature
  2. /onboarding/step1  score=16  sessions/day=5k   business=onboarding
  3. /reports/{id}      score=12  sessions/day=1.2k business=primary feature

/checkout and /login carry the highest scores but already have monitors, so they drop out of the diff. /dashboard surfaces as gap rank 1: 12 k sessions/day, a primary feature, and no monitor - exactly the silent-failure risk this skill exists to catch. Hand these three rows to synthetic-monitor-author.

Hard-reject rule

If no RUM source is available (no Datadog RUM, no Sentry Performance data, no CrUX data for the target site), halt and return:

HALT: no RUM data source available.
Supply at least one of: Datadog RUM Explorer access, Sentry Performance
transaction list, or a CrUX-eligible public origin.
Synthetic coverage gap analysis requires real usage data as input.

Do not estimate journey volume from gut feel or static sitemap inspection. Gap prioritization without usage data produces a monitor list biased by developer assumptions rather than actual user behavior.

Anti-patterns

Limitations

• Datadog RUM session retention is 30 days per Datadog RUM Search; seasonally low periods (holiday shutdowns, pre-launch) may undercount volume. Use a representative date range.
• Sentry samples high-volume transactions by default; throughput numbers from Sentry Transaction Summary (TPM/TPS) reflect sampled traces. Check your SDK sample rate before interpreting absolute volumes.
• CrUX 28-day rolling window smooths spikes; a newly launched page with three weeks of traffic may not yet appear. Use Datadog RUM or GA4 for recently launched pages.
• Business value scores are editorial. The 1-5 table in Step 2 is a starting point, not a formula. Align with product stakeholders before the first run and document the agreed values in the gap report header.
• This skill produces a gap list, not monitors. Authoring the actual monitor configs - script, cadence, assertions, alert thresholds - is the responsibility of synthetic-monitor-author.

References

• references/rum-source-queries.md - per-source Step 1 queries for Datadog RUM, Sentry Performance, and GA4 + CrUX.
• Datadog RUM Explorer - event types, grouping, count aggregation.
• Datadog RUM Search - key:value facet syntax, six core event types.
• Datadog RUM Dashboards - Performance Overviews, Usage, Testing and Deployment.
• Sentry Transaction Summary - TPM/TPS throughput, p50/p75/p95, User Misery.
• CrUX docs - dataset overview, access methods (API, BigQuery, PageSpeed Insights).
• CrUX methodology - eligibility criteria, public discoverability requirement.
• web.dev Core Web Vitals - LCP/INP/CLS definitions, good/needs-improvement/poor thresholds, 75th percentile assessment standard.
• synthetic-monitor-author - downstream skill: authors the monitor config for each gap identified here.
• ISTQB Glossary V4.7.1 https://glossary.istqb.org/en_US/term/shift-right - defines shift right as "A test approach to test a system continuously in production"; this skill feeds that approach by ensuring synthetic coverage reflects real production usage patterns.