CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

adobe/stardust

Redesign an existing website to make it better. Built on top of impeccable.

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

Overview
Quality
Evals
Security
Files

crawl.mjsskills/extract/scripts/

#!/usr/bin/env node
/**
 * crawl.mjs — reference Playwright crawler for stardust:extract.
 *
 * Solves two stardust multitest findings:
 *   #4  extract ships no runnable crawler — every migration re-implements the
 *       Playwright recipe by hand (expensive, inconsistent). This is the bundled
 *       `extract/scripts/crawl.mjs` the recipe always implied.
 *   #7  capture hardening — the hand-rolled crawls captured hidden / transient /
 *       modal DOM as real content (consent banners and "temporarily unavailable"
 *       overlays became headings; AJAX-modal detail pages captured byte-identical
 *       to their listing; SPA shells captured a tracking pixel + global h1 as a
 *       "page"). This crawler filters those at capture time.
 *
 * It implements the CORE of reference/playwright-recipe.md (browser config +
 * bot-management fallback, consent dismissal, wait+scroll, the capture list,
 * response validation) plus the finding-#7 hardening below. The recipe remains
 * the authoritative field spec; extend the in-page capture() to match it fully.
 *
 * Hardening (#7), all applied inside the page context:
 *   - VISIBILITY FILTER: headings/body/CTAs skip nodes that are display:none,
 *     visibility:hidden, aria-hidden, [hidden], or off-screen / zero-area.
 *   - INTERSTITIAL/ERROR heuristic: nodes matching known consent / language-gate
 *     / "temporarily unavailable" patterns are dropped from content and counted
 *     in `_filtered`.
 *   - MODAL/AJAX capture: [role=dialog] / .modal / [aria-modal] containers are
 *     read via textContent even when display:none (XHR-populated detail), so a
 *     URL-addressable modal route is not captured as its listing page.
 *   - TRACKING-PIXEL = zero media: a lone off-origin <=2px img doesn't count as
 *     "has media" (so the low-media flag fires on an SPA shell).
 *   - SUBSTANCE check: a page with <2 distinct in-main headings AND tiny main
 *     innerText AND no real media is flagged `spaShellSuspect`.
 *   - DUPLICATE check (cross-page, after the crawl): a page whose main-content
 *     hash equals another page's is flagged `duplicateOf` (catches detail==listing).
 *     Attribution is deterministic by discovery order: the earliest-queued page
 *     per hash is canonical, regardless of pool completion order.
 *   - RENDERED DOM: the settled page's `page.content()` is saved verbatim as
 *     <out>/pages/<slug>.html next to the JSON (path in the record's
 *     `renderedHtml` field). Capture once, parse offline: importers and sibling
 *     generators iterate their extraction against this artifact (free,
 *     reproducible, and provenance) instead of re-running live probes per
 *     selector guess. Live probes stay for what the static DOM cannot answer
 *     (geometry, computed styles).
 *   - SCREENSHOT: a full-page PNG per page under <out>/assets/screenshots/<slug>.png
 *     (the page is measured first — screenshotPage: capped at 30000px, viewport-only when
 *     the page's own CSS overflowed to Chrome's layout maximum; mode in _signals.screenshotMode,
 *     relative path in the page record's `screenshot` field) — feeds the extract
 *     SKILL.md Phase 2.5 vision gate. A page with no screenshot or a layout overflow is a
 *     `[crawl] WARN` line and an entry in _crawl-log.json#crawl.degraded with the reason.
 *   - LOCALE-ROOT CAPTURE GAPS (cross-page, after the crawl): for every captured
 *     locale root — a one- or two-segment path of locale codes, such as /en, /fr-ca,
 *     /ca/en — the same-origin links ON that page under its own path are listed and
 *     the ones the crawl never captured are written to _crawl-log.json#captureGaps
 *     ({ roots: [{ root, slug, linked, uncaptured[] }], detail }) and summarised on
 *     stderr. `detail` is the ready ledger string ("uncaptured first-level targets:
 *     ca/en 15, fr/fr 0") for the extract `end` line, so the owner sees the gap
 *     before Phase 2. Two recorded hands-off runs captured a locale root whose
 *     15-page subtree discovery never listed, and learned it six hours later from
 *     the rollout link audit (15 × 404, then repointed to the source).
 *
 * Usage:
 *   node crawl.mjs --url https://example.com [--pages a,b,c] [--max 25] [--link-first] \
 *     [--out stardust/current] [--wait medium] [--no-consent-dismiss] \
 *     [--concurrency 4] [--dynamics] [--page-timeout 60]
 *     --url <url>             required; the entry page — its origin scopes discovery
 *     --pages a,b,c           crawl exactly these paths/URLs (resolved against --url; the entry
 *                             is added) instead of discovering via sitemap / same-origin nav links
 *     --max <n>               discovery cap (default 25; 0 = no cap; listed --pages are never dropped)
 *     --link-first            the user named the page count: entry page, then its own same-origin page
 *                             links in document order, then the other discovered pages, up to --max
 *     --out <dir>             output root (default stardust/current)
 *     --wait fast|medium|slow settle after load: 1200 / 2500 / 5000 ms (default medium)
 *     --no-consent-dismiss    do not auto-dismiss consent overlays before capture
 *     --concurrency <n>       parallel browser contexts (default 4)
 *     --dynamics              also record each page's dynamic surface (network endpoints, forms,
 *                             hydration hints) — set by the calling skill, off by default
 *     --page-timeout <s>      per-page deadline (default 60; 0 = none): a page that never settles is
 *                             recorded as TimeoutError instead of eating the run's cap
 *
 * Browser and bot walls: real Chrome headless (channel: chrome — the installed browser's own identity,
 * no launch flags) first, bundled Chromium only when Chrome is missing (technique chrome |
 * chromium-fallback). A wall is NAMED within seconds, never captured as content: an edge challenge
 * (cf-mitigated, or a 403/429/503 with a Cloudflare / Akamai / F5 / Imperva signature, a 403 from
 * CloudFront / AWS WAF), or a 2xx VERIFICATION INTERSTITIAL served as a normal page ("One moment,
 * please…" reloading itself, "Just a moment…", CloudFront's "Request blocked" body — read from the
 * page, not the headers: this script's title rule, then the vendor list in
 * stardust/scripts/browser.mjs). On the ENTRY page → one relaunch (technique chrome-escalated;
 * the existing launch flags and webdriver spoof stay on, STARDUST_STEALTH=0 turns them off;
 * STARDUST_HEADED_WINDOW=1 is not read here) with the
 * challenge-solve window, then a fatal `bot-management … not cleared` error naming the wall; on an
 * inner page → the page is recorded as failed with errorClass BotChallenge and the marker. Live
 * navigations to one host are paced across processes — STARDUST_LIVE_PACE_MS apart (default 1000; 0 disables),
 * at most STARDUST_LIVE_MAX_PARALLEL in flight (default 2), backing off on bot signals — by the shared throttle
 * (runLiveNavigation) in stardust/scripts/browser.mjs.
 *
 * Writes capture artifacts under --out. The shared helper (stardust/scripts/browser.mjs)
 * keeps its repository-scoped throttle and browser state under OS temp:
 *   pages/<slug>.json              one record per captured page — slug, url, finalUrl, title,
 *                                  description, og, headings, body, codeBlocks, ctas, links,
 *                                  media {imgs, allImgCount, cssBackgrounds, modals, videos,
 *                                  iframes}, customProps (CSS custom properties resolved on
 *                                  :root), _signals, _provenance, renderedHtml, screenshot
 *                                  (+ `dynamic` with --dynamics; `_signals.duplicateOf` marks a
 *                                  page whose main content hashes equal to an earlier page's)
 *   pages/<slug>.html              the settled rendered DOM, verbatim (parse offline — never re-scrape)
 *   assets/screenshots/<slug>.png  full-page screenshot (`_signals.screenshotMode`: fullPage |
 *                                  capped | viewport | failed; `screenshotReason`, `pageSize`,
 *                                  `layoutOverflow` say why)
 *   assets/favicon.<ext>           the site icon from <link rel~=icon> or /favicon.ico, <ext> from
 *                                  its content-type — the ONLY media file this crawler downloads
 *   _crawl-log.json                discovery technique + page count, consent mode, favicon result,
 *                                  per-URL failures, `crawl.degraded` (pages captured without a
 *                                  usable screenshot, or with a layout overflow), slash retries, `captureGaps` (locale roots,
 *                                  above) (+ `dynamicSurface` with --dynamics); merged into an
 *                                  existing log, never replaced
 * Not captured here: page images and videos (recorded as URLs under `media`, never downloaded),
 * stylesheets, font files, the logo and the rest of the brand surface — those belong to the
 * extract skill's Phase 3 (brand-surface extraction), not to the crawl; do not look for them
 * under --out after a crawl. Progress lines go to stderr. Exit 0 on completion, 2 on a fatal
 * error (unknown or missing arguments included).
 *
 * Needs playwright importable from the project (see extract/SKILL.md Setup —
 * `npm i -D playwright` or the Playwright MCP server; the `npx playwright`
 * availability probe alone does NOT make the ESM module importable).
 */
import { mkdir, writeFile, readFile } from 'node:fs/promises';
import { existsSync, readFileSync, realpathSync } from 'node:fs';
import path from 'node:path';
import crypto from 'node:crypto';
import { fileURLToPath } from 'node:url';

// Compare by real path: a symlinked checkout or temp dir must not turn the CLI into a silent no-op — and
// a test that imports the pure helpers below (captureGaps, localeRootPath) must not start a crawl.
const SELF = fileURLToPath(import.meta.url);
function safeRealpath(p) { try { return realpathSync(p); } catch { return p; } }
const IS_MAIN = Boolean(process.argv[1]) && SELF === safeRealpath(process.argv[1]);

// --help prints this file's usage header, so an agent never reads the source to learn the flags.
if (IS_MAIN && (process.argv.includes('--help') || process.argv.includes('-h'))) {
  const src = readFileSync(new URL(import.meta.url), 'utf8');
  const header = src.match(/\/\*\*[\s\S]*?\*\//);
  console.log(header ? header[0].replace(/^\/\*\*\s*|\s*\*\/$/g, '').replace(/^\s*\* ?/gm, '').trim() : 'no usage header');
  process.exit(0);
}

const WAIT_MS = { fast: 1200, medium: 2500, slow: 5000 };

// One context config for probe, headed fallback, and workers — a tweak (locale,
// UA, colorScheme) must land everywhere or discovery renders under different
// conditions than capture. Viewport here also saves a per-page CDP round-trip.
const CRAWL_CONTEXT = { reducedMotion: 'reduce', viewport: { width: 1440, height: 900 } };

const sharedHookWarningHosts = new Set();
function warnSharedHook(url, error) {
  let host;
  try { host = new URL(url).host; } catch { host = 'unknown host'; }
  if (sharedHookWarningHosts.has(host)) return;
  sharedHookWarningHosts.add(host);
  console.error(`[crawl] WARNING: shared live-session bookkeeping unavailable for ${host}: ${String(error && error.message ? error.message : error).split('\n')[0]} — continuing without it`);
}
async function callSharedHook(hook, args, url) {
  try { return await hook(...args); }
  catch (error) { warnSharedHook(url, error); return false; }
}

/** The front door's live-session helpers; a hook that throws is reported once per host and the crawl goes on. */
export function crawlerLiveBindings(shared) {
  return {
    newLiveContext: shared.newLiveContext,
    runLiveNavigation: async (page, url, navigate) => {
      let operation;
      const run = () => {
        if (!operation) {
          operation = Promise.resolve().then(navigate);
          operation.catch(() => {});
        }
        return operation;
      };
      try {
        return await shared.runLiveNavigation(page, url, run);
      } catch (error) {
        if (!operation) { warnSharedHook(url, error); return run(); }
        const response = await operation; // the navigation's own rejection still escapes unchanged
        warnSharedHook(url, error);
        return response;
      }
    },
    persistLiveStorageState: (page, url) => callSharedHook(shared.persistLiveStorageState, [page, url], url),
    recordLiveBotSignal: (url) => callSharedHook(shared.recordLiveBotSignal, [url], url),
    recordLiveNavigationSuccess: (url) => callSharedHook(shared.recordLiveNavigationSuccess, [url], url),
    liveResponseSignalRecorded: shared.liveResponseSignalRecorded,
  };
}

/** The discovery cap from --max: 0 means no cap (--all, --prep), a missing or bad value the default 25. Pure. */
export const maxPages = (v) => (String(v).trim() === '0' ? Infinity : Math.max(1, Math.floor(+v) || 25));
/** The per-page deadline from --page-timeout <s> in ms: 0 means none, a missing or bad value the default 60 s. Pure. */
export const PAGE_TIMEOUT_MS_DEFAULT = 60000;
export function pageTimeoutMs(v) {
  if (v === undefined || v === null || String(v).trim() === '') return PAGE_TIMEOUT_MS_DEFAULT;
  const n = Number(v);
  return Number.isFinite(n) && n >= 0 ? Math.floor(n * 1000) : PAGE_TIMEOUT_MS_DEFAULT;
}

export function parseArgs(argv) {
  const a = { out: 'stardust/current', max: 25, linkFirst: false, wait: 'medium', consent: true, concurrency: 4, dynamics: false, pageTimeoutMs: PAGE_TIMEOUT_MS_DEFAULT };
  for (let i = 2; i < argv.length; i += 1) {
    const k = argv[i];
    if (k === '--url') a.url = argv[(i += 1)];
    else if (k === '--pages') a.pages = argv[(i += 1)].split(',').map((s) => s.trim()).filter(Boolean);
    else if (k === '--out') a.out = argv[(i += 1)];
    else if (k === '--max') a.max = maxPages(argv[(i += 1)]);
    else if (k === '--link-first') a.linkFirst = true;
    else if (k === '--wait') a.wait = argv[(i += 1)];
    else if (k === '--no-consent-dismiss') a.consent = false;
    else if (k === '--concurrency') a.concurrency = Math.max(1, +argv[(i += 1)] || 4);
    else if (k === '--dynamics') a.dynamics = true; // migration-bound: set by prepare-migration / replica / migrate, never by default
    else if (k === '--page-timeout') a.pageTimeoutMs = pageTimeoutMs(argv[(i += 1)]);
    else throw new Error(`unknown arg: ${k}`);
  }
  if (!a.url) throw new Error('--url is required');
  a.origin = new URL(a.url).origin;
  return a;
}

const slugify = (u) => {
  const { pathname } = new URL(u);
  const s = pathname.replace(/^\/|\/$/g, '').replace(/[^a-z0-9]+/gi, '-').toLowerCase();
  return s || 'index';
};

// Slugs key the output FILES (pages/<slug>.json, screenshots/<slug>.png), but
// distinct pages can collide on one slug: dedupeKey keeps url.search (so
// /p?a=1 and /p?a=2 are two pages) while slugify reads pathname only, and
// /about-us vs /about/us flatten identically. Without disambiguation two
// concurrent workers write the same files (silent last-writer-wins) and the
// duplicate post-pass can mark a page duplicateOf itself. Assign slugs once,
// up front: first claimant keeps the clean slug, later distinct pages get a
// deterministic -<hash4> suffix.
function assignSlugs(urls) {
  const bySlug = new Map(); // slug -> dedupeKey of first claimant
  return urls.map((u) => {
    const base = slugify(u);
    const key = dedupeKey(u);
    if (!bySlug.has(base)) { bySlug.set(base, key); return base; }
    if (bySlug.get(base) === key) return base; // same page (shouldn't recur post-dedupe)
    const suffix = crypto.createHash('sha1').update(key).digest('hex').slice(0, 4);
    const alt = `${base}-${suffix}`;
    if (!bySlug.has(alt)) bySlug.set(alt, key);
    return alt;
  });
}

// ---- the shared browser module (stardust/scripts/browser.mjs, the front door), beside this script ----
// Three layouts: plugin tree, project copy (stardust/scripts/<skill>/), flat (stardust/scripts/). It lends
// the crawler the browser, the consent vendor list, the denial-page reader, the per-host pacing and the
// deadline helper. Without it the crawler stops (exit 2, its fatal-error code); a hook that fails at run time is tolerated below.
const BROWSER = ['../../stardust/scripts/browser.mjs', '../stardust/browser.mjs', './stardust/browser.mjs']
  .map((p) => new URL(p, import.meta.url)).find((u) => existsSync(u));
let liveSessionModule;
async function liveSession() {
  if (liveSessionModule === undefined) {
    if (!BROWSER) {
      console.error('crawl: browser.mjs not found (looked in ../../stardust/scripts/, ../stardust/ and ./stardust/). Copy the master skill\'s scripts dir to stardust/scripts/stardust/ (skills/stardust/SKILL.md § Setup).');
      process.exit(2);
    }
    liveSessionModule = await import(BROWSER.href);
  }
  return liveSessionModule;
}

// ---- the browser: real Chrome headless first, its own identity; bundled Chromium only when Chrome is missing ----
// The default tier presents itself as what it is — the installed Chrome, headless, under automation: no user agent
// is set, no launch flag is passed; the real binary is what clears TLS/H2-fingerprint blocks (not a window — the
// container has no display). The escalation below keeps the launch flags and the navigator.webdriver spoof the
// crawler always had; STARDUST_STEALTH=0 turns them off.
const stealthEnabled = () => process.env.STARDUST_STEALTH !== '0';
const STEALTH_ARGS = ['--disable-blink-features=AutomationControlled'];
async function launchWithFallback() {
  const { openBrowser, browserTier } = await liveSession();
  const browser = await openBrowser({ mode: 'chrome' });
  return { browser, technique: browserTier(browser) };
}
// The one escalation after an entry-page block: a fresh real Chrome, window-free (the container has no display),
// with the existing launch flags unless STARDUST_STEALTH=0. A launch failure is one stderr line and null — the
// crawl continues with the browser it has.
async function launchEscalated() {
  const stealth = stealthEnabled();
  const opts = { headless: true, channel: 'chrome', ...(stealth ? { args: [...STEALTH_ARGS], ignoreDefaultArgs: ['--enable-automation'] } : {}) };
  const { openBrowser } = await liveSession();
  try {
    return await openBrowser({ mode: 'plain', launchOptions: opts, tier: 'chrome-escalated' });
  } catch (e) {
    console.error(`[crawl] WARN real Chrome relaunch failed (${String(e.message).split('\n')[0]}) — continuing with the browser already running`);
    return null;
  }
}
// A context factory so the webdriver spoof lands on EVERY context (probe + workers) once the run escalated — the
// challenge re-fires per context (no cross-context cookie sharing). Before the escalation, and under
// STARDUST_STEALTH=0, no init script: navigator.webdriver stays true.
async function newContext(browser, stealth, stateUrl) {
  const { newLiveContext } = crawlerLiveBindings(await sharedLiveSession());
  const ctx = await newLiveContext(browser, { ...CRAWL_CONTEXT, stateUrl });
  if (stealth) {
    await ctx.addInitScript(() => {
      Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
    });
  }
  return ctx;
}
// Network-level fingerprint reject: navigation THROWS before any JS runs.
export function isFingerprintBlock(err) {
  const m = String(err && err.message || err);
  return /ERR_HTTP2_PROTOCOL_ERROR|ERR_QUIC_PROTOCOL_ERROR|ERR_CONNECTION_RESET|net::ERR/.test(m);
}
// Bot-management CHALLENGE / block: navigation SUCCEEDS (domcontentloaded fires,
// no throw) but the response is a 403/429/503 interstitial, not the page. The
// original fallback only caught isFingerprintBlock() throws, so a Cloudflare
// managed challenge (cf-mitigated: challenge, HTTP 403) sailed past the probe
// and only blew up at capture-time as a fatal HTTPError. Validate the RESPONSE,
// not just DOM-ready. Returns the marker naming the signature, or null — the same
// rules as live-session.mjs challengeMarker (keep the two in step).
export function challengeMarker(resp) {
  if (!resp) return null;
  const status = resp.status();
  const h = resp.headers();
  // Cloudflare stamps this header specifically on managed/JS-challenge responses.
  if ((h['cf-mitigated'] || '').toLowerCase() === 'challenge') return 'cf-mitigated: challenge';
  // AWS WAF stamps its action (captcha / challenge) on the response it intercepts, whatever the status.
  if (h['x-amzn-waf-action']) return `HTTP ${status} + CloudFront/AWS WAF edge signature (x-amzn-waf-action: ${h['x-amzn-waf-action']})`;
  // A hard 403/429/503 that ALSO carries an edge/CDN signature is an edge
  // interstitial (Cloudflare / Akamai / F5 / Imperva / CloudFront) — the escalation is
  // the correct response regardless of vendor. Requiring the edge signature (not the
  // bare status) is deliberate: the marker gates clearChallenge() on EVERY page, so a
  // legitimate app-level 403 (e.g. an auth-gated deep page with no CDN header) must
  // fail fast, not eat the ~12s challenge-solve retry loop.
  const server = (h.server || '').toLowerCase();
  if (status === 403 || status === 429 || status === 503) {
    if (h['cf-ray'] || server.includes('cloudflare')) return `HTTP ${status} + Cloudflare edge signature (cf-ray/server)`;
    if (h['x-akamai-transformed'] || server.includes('akamai')) return `HTTP ${status} + Akamai edge signature`;
    if (server.includes('big-ip') || server.includes('bigip') || server.includes('imperva') || h['x-iinfo']) return `HTTP ${status} + F5/Imperva edge signature`;
    // CloudFront answering 403 itself after a burst of requests ("The request could not be satisfied. Request blocked.")
    // is the AWS WAF block (recorded on a CloudFront-fronted site); an origin 403 relayed by CloudFront carries the same
    // headers, so the label names the edge, not a verdict on the origin.
    if (status === 403 && (server.includes('cloudfront') || (h['x-cache'] || '').toLowerCase().includes('error from cloudfront'))) return 'HTTP 403 + CloudFront/AWS WAF edge signature (server/x-cache)';
    // no edge signature — treat as a genuine app-level status, not a challenge.
  }
  // Akamai escalates to HTTP 400 {"result":"Bad Request"} after a burst of probes.
  if (status === 400 && (server.includes('akamai') || h['x-akamai-transformed'])) return 'HTTP 400 + Akamai edge signature (bot-management escalation)';
  return null;
}
export const isChallengeResponse = (resp) => challengeMarker(resp) !== null;

// ---- the 2xx verification interstitial: a wall served as a normal page ----
// A site served "One moment, please…" (HTTP 200, reloading itself every few seconds) to every client and the crawler
// waited out the run's whole cap without naming it. The headers say nothing; the PAGE is read. The inline rule below
// reads first: a <title> or <h1> that names a wall on a page under 2 KB of visible text, or one that reloads itself
// (<meta http-equiv=refresh> / location.reload) under such a title. The shared reader (live-session pageDenial: vendor
// signatures, the tiny one-error-heading page) reads after it.
// "verify" alone is no wall word: an age or e-mail verification page is a page, and a small single-page-app entry page
// under such a title would abort the crawl; the verification interstitials name the visitor ("verify you are human",
// "verifying your request / browser", "human verification", "checking your browser").
export const WALL_TITLE = /one moment|just a moment|verify (?:that )?you(?:'re| are)|verifying (?:(?:that )?you(?:'re| are)|your (?:request|browser|connection))|human verification|are you (?:a )?human|checking your browser|pardon our interruption|access denied|request blocked|attention required/i;
export const WALL_TEXT_MAX_BYTES = 2048;
/* eslint-disable no-undef */
/** page.evaluate(wallProbeInPage) → { title, h1, bytes, selfReload } (self-contained: playwright serialises it). */
export function wallProbeInPage() {
  const collapse = (s) => String(s || '').replace(/\s+/g, ' ').trim();
  const text = collapse(document.body ? document.body.innerText : '');
  const h1 = document.querySelector('h1');
  const metaRefresh = Boolean(document.querySelector('meta[http-equiv="refresh" i]'));
  const scriptReload = [...document.scripts].some((s) => /location\.reload\(|location\.href\s*=\s*(?:window\.)?location|window\.location\s*=\s*window\.location/.test(s.textContent || ''));
  return { title: document.title, h1: h1 ? collapse(h1.innerText) : '', bytes: new TextEncoder().encode(text).length, selfReload: metaRefresh || scriptReload };
}
/* eslint-enable no-undef */
/** The inline wall marker of a wallProbeInPage probe, or null (pure). */
export function inlineWallMarker(probe) {
  if (!probe || typeof probe !== 'object') return null;
  const title = String(probe.title || '').replace(/\s+/g, ' ').trim();
  const h1 = String(probe.h1 || '').replace(/\s+/g, ' ').trim();
  const hit = [title, h1].find((t) => t && WALL_TITLE.test(t));
  if (!hit) return null;
  const small = Number.isFinite(probe.bytes) && probe.bytes < WALL_TEXT_MAX_BYTES;
  if (!small && !probe.selfReload) return null;
  return `interstitial page — "${hit.slice(0, 80)}"${probe.selfReload ? ' (self-reloading)' : ''}`;
}
// One wall read's deadline: page.evaluate has no timeout of its own, and a renderer stuck in a busy loop never answers it.
const WALL_PROBE_MS = 10000;
// → { marker, skipped }: this script's own rule first (wallProbeInPage + inlineWallMarker — a wall named by its title or
// h1 on a small or self-reloading page), then the shared reader (live-session pageDenial: the vendor signature list
// and the one-error-heading rule). A page no reader could read is `skipped` with the error's first line (the caller
// logs it — never a silent pass, never a false wall).
async function pageWall(page) {
  const { withDeadline, pageDenial } = await liveSession();
  let skipped = null;
  try {
    const marker = inlineWallMarker(await withDeadline(page.evaluate(wallProbeInPage), WALL_PROBE_MS, 'wall probe'));
    if (marker) return { marker, skipped: null };
  } catch (e) {
    skipped = String(e && e.message ? e.message : e).split('\n')[0];
  }
  const shared = await pageDenial(page);
  if (shared.marker) return shared;
  return { marker: null, skipped: skipped && shared.skipped ? skipped : null };
}
const ENTRY_SETTLE_MS = 1200;
/**
 * The entry-page probe: navigate, then classify — null (the page), or { kind, marker } with kind 'fingerprint' (the
 * goto threw a network-level reject), 'challenge' (an edge signature on the response) or 'interstitial' (a 2xx wall
 * read from the page after a short settle). With `solve` (the escalated probe) a challenge gets clearChallenge's
 * wait+reload window and an interstitial up to three settled reloads before it counts. Any other goto error throws.
 */
export async function probeEntry(page, url, { timeout = 30000, solve = false } = {}) {
  let resp;
  const { goto } = await liveSession();
  try {
    resp = await sharedNavigate(page, url, () => goto(page, url, { waitUntil: 'domcontentloaded', timeout }));
  } catch (err) {
    // a network-level reject is not a pacing matter (the relaunch answers it) — no shared signal for it
    if (isFingerprintBlock(err)) return { kind: 'fingerprint', marker: String(err.message || err).split('\n')[0] };
    throw err;
  }
  if (solve) resp = await clearChallenge(page, resp); // records the challenge signal itself, once per response
  const edge = challengeMarker(resp);
  if (edge) {
    // the first probe's verdict slows every process pacing this host; a 429 the shared navigation already counted is
    // not counted twice, and the escalated probe's clearChallenge has recorded its own
    if (!solve) await recordSharedBotSignal(url, resp).catch(() => {});
    return { kind: 'challenge', marker: edge };
  }
  if (!resp || resp.status() >= 400) return null; // an app-level status: not a wall; the capture reports it as HTTPError
  await page.waitForTimeout(ENTRY_SETTLE_MS);
  let wall = await pageWall(page);
  for (let attempt = 0; solve && wall.marker && attempt < 3; attempt += 1) {
    await page.waitForTimeout(4000);
    const reloaded = await sharedNavigate(page, url, () => page.reload({ waitUntil: 'domcontentloaded', timeout })).catch(() => null);
    if (!reloaded || challengeMarker(reloaded) || reloaded.status() >= 400) break; // not cleared
    await page.waitForTimeout(ENTRY_SETTLE_MS);
    const next = await pageWall(page);
    if (!next.skipped) wall = next; // only a verdict replaces the confirmed wall (fail closed)
  }
  if (wall.skipped) console.error(`[crawl] wall check skipped on the entry page: ${wall.skipped}`);
  if (!wall.marker) return null;
  await recordSharedBotSignal(url, resp).catch(() => {}); // a 2xx wall: no header layer counted it
  return { kind: 'interstitial', marker: wall.marker };
}

// ---- the per-page deadline (--page-timeout): a page that never settles is a TimeoutError, not the run's cap ----
async function withPageDeadline(promise, ms) {
  if (!(ms > 0)) return promise;
  const { withDeadline } = await liveSession();
  try {
    return await withDeadline(promise, ms, 'page capture');
  } catch (e) {
    if (e && e.name === 'DeadlineError') throw Object.assign(new Error(`page deadline ${ms / 1000} s exceeded (--page-timeout)`), { errorClass: 'TimeoutError' });
    throw e;
  }
}
// Cloudflare's non-interactive managed challenge serves the 403/503 interstitial,
// runs its JS, sets a clearance cookie, then the real page becomes reachable.
// Wait for that window and reload to pick up the cookie before treating the
// status as a hard failure. No-op for a normal 200 (isChallengeResponse false),
// so zero overhead on the common path. A challenged page is ONE shared bot signal,
// however many reloads stay challenged: the reloads ask for the same page 4 s apart
// and are the same wall, not new evidence (counted per reload, one walled inner
// page took its host to the top of the backoff ladder within a single crawl). A
// response the transport layer already counted (a marked 429) is the page's signal.
export async function clearChallenge(page, resp, {
  recordBotSignal = recordSharedBotSignal, navigate = sharedNavigate, signalRecorded,
} = {}) {
  const alreadyRecorded = signalRecorded || (await sharedLiveSession()).liveResponseSignalRecorded;
  let signalled = false;
  const recordSignal = async () => {
    if (signalled || !isChallengeResponse(resp)) return;
    signalled = true;
    if (!alreadyRecorded(resp)) await callSharedHook(recordBotSignal, [page.url()], page.url());
  };
  await recordSignal();
  for (let attempt = 0; attempt < 3 && isChallengeResponse(resp); attempt += 1) {
    await page.waitForTimeout(4000);
    const reloaded = await navigate(page, page.url(), () => page.reload({ waitUntil: 'domcontentloaded', timeout: 45000 }))
      .catch(() => null);
    if (reloaded) { resp = reloaded; await recordSignal(); }
  }
  return resp;
}

// ---- URL normalization: one canonical form for entry, --pages, sitemap, BFS ----
// resolve against base, strip hash, keep query, normalize trailing slash
// (non-root paths lose it) so `/about`, `/about/` and `/about#team` dedupe.
function normalizeUrl(u, base) {
  const url = new URL(u, base);
  url.hash = '';
  // Keep the source's trailing-slash form VERBATIM (stardust-style e2e finding):
  // static hosts commonly serve /docs/ as 200 and /docs as 404 with NO redirect
  // between the variants, so rewriting the fetched URL turns sitemap-declared
  // pages into 404s. Dedupe happens by slash-stripped KEY (dedupeKey below),
  // never by rewriting the URL we fetch.
  return url.href;
}
// slash-insensitive identity for dedupe: /about, /about/ and /about#x are one page.
function dedupeKey(href) {
  const url = new URL(href);
  return url.origin + url.pathname.replace(/\/+$/, '') + url.search;
}

// ---- locale-root capture gaps (post-pass; pure, exported for tests) ----
// A locale root is a captured page whose path is one or two segments of locale codes — `/en`,
// `/fr-ca`, `/de_DE`, `/ca/en` (region, then language). Its first-level targets are the same-origin
// links ON that page that lie under its own path (query, hash and trailing slash folded; assets,
// mailto:/tel: and other hosts dropped; a `www.` prefix folded). Those the crawl never captured
// (no record with that url or finalUrl) are the gaps. Two recorded hands-off runs captured a locale
// root while its 15-page subtree, linked from that very page, never entered the inventory.
const LOCALE_SEG = /^[a-z]{2,3}(?:[-_][a-z]{2,4})?$/i;
const ASSET_EXT = /\.(?:css|js|mjs|json|xml|txt|pdf|zip|gz|ico|png|jpe?g|gif|svg|webp|avif|mp[34]|webm|woff2?|ttf|otf|eot)$/i;
export function localeRootPath(href) {
  let u;
  try { u = new URL(href); } catch { return null; }
  const segs = u.pathname.split('/').filter(Boolean);
  if (!segs.length || segs.length > 2 || !segs.every((s) => LOCALE_SEG.test(s))) return null;
  return `/${segs.join('/')}`;
}
const hostKey = (u) => u.host.toLowerCase().replace(/^www\./, '');
const pathKey = (u) => u.pathname.replace(/\/+$/, '') || '/';
// a record's `links` is the flat href list capture() writes; the schema's { internal, external } shape is read too
const linkList = (links) => (Array.isArray(links) ? links : links && typeof links === 'object' ? [...(links.internal || []), ...(links.external || [])] : []);
export function captureGaps(records) {
  const captured = new Set();
  for (const r of records) {
    for (const h of [r.url, r.finalUrl]) { try { const u = new URL(h); captured.add(`${hostKey(u)}${pathKey(u)}`); } catch { /* absent or not a URL */ } }
  }
  const roots = [];
  for (const r of records) {
    const root = localeRootPath(r.url);
    if (!root) continue;
    const base = new URL(r.url);
    const targets = new Set();
    for (const raw of linkList(r.links)) {
      let u;
      try { u = new URL(raw, r.url); } catch { continue; }
      if (!/^https?:$/.test(u.protocol) || hostKey(u) !== hostKey(base)) continue;
      const p = pathKey(u);
      if (!p.startsWith(`${root}/`) || ASSET_EXT.test(p)) continue;
      targets.add(p);
    }
    const uncaptured = [...targets].filter((p) => !captured.has(`${hostKey(base)}${p}`)).sort();
    roots.push({ root, slug: r.slug, linked: targets.size, uncaptured });
  }
  roots.sort((a, b) => a.root.localeCompare(b.root));
  const detail = roots.length
    ? `uncaptured first-level targets: ${roots.map((r) => `${r.root.slice(1)} ${r.uncaptured.length}`).join(', ')}`
    : 'uncaptured first-level targets: none (no locale root captured)';
  return { roots, detail };
}

/**
 * The page list: entry first, normalized dedupe, capped at max. With linkFirst (the user named N), the entry
 * page's links follow in document order, then the discovered list, both kept to same-origin non-asset pages;
 * without it, the discovered list follows the entry directly. Unparsable hrefs are skipped and returned in
 * `skipped`. Pure. Returns { urls, skipped }.
 */
export function selectPages({ entry, origin, linked, discovered, max, linkFirst }) {
  const skipped = [];
  const parse = (list) => list.flatMap((h) => {
    try { return [normalizeUrl(h, origin)]; } catch { skipped.push(h); return []; }
  });
  const page = (u) => { const url = new URL(u); return url.origin === origin && !ASSET_EXT.test(url.pathname); };
  const candidates = linkFirst ? [...parse(linked), ...parse(discovered)].filter(page) : parse(discovered);
  const seen = new Set(); const urls = [];
  for (const u of [entry, ...candidates]) {
    const k = dedupeKey(u);
    if (!seen.has(k)) { seen.add(k); urls.push(u); }
  }
  return { urls: urls.slice(0, max), skipped };
}

// ---- discovery: explicit pages > sitemap (validated) > BFS from nav ----
export async function discover(args, page) {
  const entry = normalizeUrl(args.url);
  // explicit --pages: NEVER drop a listed page. The entry URL is ADDED on top
  // (the effective cap grows by one when needed); if the total still exceeds
  // --max, warn instead of silently evicting a requested page.
  if (args.pages) {
    const seen = new Set();
    const listed = args.pages.map((p) => normalizeUrl(p, args.url))
      .filter((u) => { const k = dedupeKey(u); if (seen.has(k)) return false; seen.add(k); return true; });
    const entryKey = dedupeKey(entry);
    const urls = listed.some((u) => dedupeKey(u) === entryKey) ? listed : [entry, ...listed];
    if (urls.length > args.max) {
      console.error(`[crawl] WARN --pages lists ${listed.length} page(s); with the entry URL the total is ${urls.length}, exceeding --max ${args.max} — crawling all of them (explicitly listed pages are never dropped)`);
    }
    return urls;
  }
  // discovered lists (sitemap/BFS) go through selectPages; --link-first puts the entry page's links first.
  const readLinks = () => page.evaluate((origin) => [...document.querySelectorAll('a[href]')]
    .map((a) => a.href).filter((h) => h.startsWith(origin)), args.origin);
  let linked = args.linkFirst ? await readLinks() : null;
  const withEntry = (discovered) => {
    const { urls, skipped } = selectPages({
      entry, origin: args.origin, linked: linked || [], discovered, max: args.max, linkFirst: args.linkFirst,
    });
    if (skipped.length) console.error(`[crawl] WARN discovery skipped ${skipped.length} unparsable URL(s): ${skipped.join(', ')}`);
    return urls;
  };
  // sitemap.xml — but only trust it if it has >=1 <loc> (a 200-but-empty Drupal
  // sitemap must fall through to BFS — finding from a media-site run).
  for (const sm of ['/sitemap.xml', '/sitemap_index.xml']) {
    try {
      const xml = await page.evaluate(async (u) => {
        const r = await fetch(u); return r.ok ? r.text() : '';
      }, new URL(sm, args.origin).href);
      const locs = [...xml.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)].map((m) => m[1])
        .filter((u) => u.startsWith(args.origin));
      // a sitemap INDEX's <loc>s are child-sitemap .xml URLs, not pages —
      // recurse one level (capped) instead of queueing them as pages, where
      // every capture would throw ContentTypeError and discovery would
      // silently collapse to the entry page.
      const isXml = (u) => /\.xml(?:[?#]|$)/i.test(u);
      let pageLocs = locs.filter((u) => !isXml(u));
      const childMaps = locs.filter(isXml).slice(0, 8);
      if (!pageLocs.length && childMaps.length) {
        for (const child of childMaps) {
          try {
            const cx = await page.evaluate(async (u) => {
              const r = await fetch(u); return r.ok ? r.text() : '';
            }, child);
            pageLocs = pageLocs.concat([...cx.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)]
              .map((m) => m[1]).filter((u) => u.startsWith(args.origin) && !isXml(u)));
          } catch { /* skip unreadable child sitemap */ }
        }
      }
      if (pageLocs.length >= 1) return withEntry(pageLocs);
    } catch { /* fall through */ }
  }
  // BFS depth-1 from the entry page's same-origin nav links.
  linked = linked || await readLinks();
  return withEntry(linked);
}

// The shared live-session hooks (throttle, browser-state jar, bot signals) ride the same loader.
const sharedLiveSession = liveSession;
async function sharedNavigate(page, url, navigate) {
  const { runLiveNavigation } = crawlerLiveBindings(await sharedLiveSession());
  return runLiveNavigation(page, url, navigate);
}
async function persistSharedState(page, url = page.url()) {
  const { persistLiveStorageState } = crawlerLiveBindings(await sharedLiveSession());
  return persistLiveStorageState(page, url);
}
async function recordSharedBotSignal(url, response) {
  const { recordLiveBotSignal, liveResponseSignalRecorded } = crawlerLiveBindings(await sharedLiveSession());
  if (liveResponseSignalRecorded(response)) return false;
  return recordLiveBotSignal(url);
}
async function recordSharedSuccess(url) {
  const { recordLiveNavigationSuccess } = crawlerLiveBindings(await sharedLiveSession());
  return recordLiveNavigationSuccess(url);
}

// The shared consent pass (stardust/scripts/browser.mjs acceptConsent — the vendor list every
// gate instrument uses) runs first (liveSession above); the passes below stay the fallback when it misses.
async function dismissConsent(page) {
  const { acceptConsent } = await liveSession();
  const shared = await acceptConsent(page).catch(() => null);
  if (shared) console.error(`[crawl] consent dismissed via ${shared}`);
  const sels = ['#onetrust-accept-btn-handler', '.truste-button2', '[aria-label*="Accept" i]',
    'button[id*="accept" i]', 'button[class*="accept" i]'];
  let matched = Boolean(shared);
  for (const s of matched ? [] : sels) {
    const el = await page.$(s);
    if (el) { await el.click().catch(() => {}); matched = true; await page.waitForTimeout(300); break; }
  }
  // Usercentrics renders inside shadow DOM (#usercentrics-root) — regular
  // selectors can't reach it (tools-retailer e2e finding).
  const ucMatched = !matched && await page.evaluate(() => {
    const root = document.querySelector('#usercentrics-root')?.shadowRoot;
    if (root) {
      const btn = root.querySelector('[data-testid="uc-deny-all-button"], [data-testid="uc-accept-all-button"]');
      if (btn) { btn.click(); return true; }
    }
    return false;
  }).catch(() => false);
  // Text-match fallback, only when the selector pass matched NOTHING (two field
  // harvests, 2026-08: two different consent widgets — a custom
  // dialog, cookieconsent's a.cc-btn — were missed by the list above; on
  // one of them the banner baked into the ground-truth screenshot AND repeated at
  // all 7 stitch seams → 32% false pixel diff). Guards keep it from ever
  // hitting an in-content link: exact match on a short consent label (≤25
  // chars after whitespace collapse), visible, and inside a fixed/sticky or
  // high-z overlay container. Worst case = today's behavior (banner stays).
  if (!matched && !ucMatched) {
    const hit = await page.evaluate(() => {
      const LABELS = new Set(['accept', 'accept all', 'allow all', 'agree', 'ok', 'decline',
        'alle akzeptieren', 'accepter']);
      const inOverlay = (el) => {
        for (let n = el; n && n !== document.documentElement; n = n.parentElement) {
          const cs = getComputedStyle(n);
          if (cs.position === 'fixed' || cs.position === 'sticky') return true;
          if (cs.position !== 'static' && +cs.zIndex >= 100) return true;
        }
        return false;
      };
      for (const el of document.querySelectorAll('button, a, [role="button"]')) {
        const t = (el.textContent || '').replace(/\s+/g, ' ').trim();
        if (!t || t.length > 25 || !LABELS.has(t.toLowerCase())) continue;
        const cs = getComputedStyle(el);
        const r = el.getBoundingClientRect();
        if (cs.display === 'none' || cs.visibility === 'hidden' || r.width < 2 || r.height < 2) continue;
        if (!inOverlay(el)) continue;
        el.click();
        return t;
      }
      return null;
    }).catch(() => null);
    if (hit) { console.error(`[crawl] consent dismissed via text-match fallback ("${hit}")`); await page.waitForTimeout(300); }
  }
  await page.waitForTimeout(300);
  // assert: prune any consent container still present (don't leave it for capture).
  await page.evaluate(() => {
    document.querySelectorAll('#onetrust-banner-sdk, #truste-consent-track, #usercentrics-root, [class*="cookie" i][class*="banner" i], [id*="consent" i]')
      .forEach((n) => n.remove());
  });
}

// Favicon — captured on the ENTRY page in ALL modes (healthcare-site harvest, 2026-08:
// bounded --pages extracts skip Phase 3 (brand surface) where favicon capture
// otherwise lives, and deploy's favicon step then skips SILENTLY — the deployed
// site shipped the default icon). One cheap request. The fetch runs in-page so
// it inherits the context's fingerprint + cookies — bot-walled origins usually
// serve assets even when pages are challenged.
async function captureFavicon(page, args) {
  try {
    const fetchIcon = (u) => page.evaluate(async (iconUrl) => {
      try {
        const r = await fetch(iconUrl);
        if (!r.ok) return null;
        return { type: r.headers.get('content-type') || '', bytes: [...new Uint8Array(await r.arrayBuffer())] };
      } catch { return null; }
    }, u);
    const href = await page.evaluate(() => document.querySelector('link[rel~="icon" i]')?.href || null);
    const fallback = new URL('/favicon.ico', args.origin).href;
    let url = href || fallback;
    let res = await fetchIcon(url);
    // a cross-origin <link> icon (CDN-hosted) dies on the CORS-bound in-page
    // fetch even when the asset is fine — retry the same-origin /favicon.ico
    // before giving up.
    if ((!res || !res.bytes.length) && url !== fallback) { url = fallback; res = await fetchIcon(url); }
    if (!res || !res.bytes.length) return null;
    // content-type is authoritative for <ext> (a /favicon.ico path routinely
    // serves PNG); the URL path is the fallback, .ico the default.
    const extFromType = /svg/.test(res.type) ? 'svg' : /png/.test(res.type) ? 'png'
      : /jpe?g/.test(res.type) ? 'jpg' : /gif/.test(res.type) ? 'gif'
        : /webp/.test(res.type) ? 'webp' : /icon/.test(res.type) ? 'ico' : '';
    const extFromPath = (path.extname(new URL(url).pathname).slice(1) || '').toLowerCase();
    const ext = extFromType || extFromPath || 'ico';
    await mkdir(path.join(args.out, 'assets'), { recursive: true });
    const file = `assets/favicon.${ext}`;
    await writeFile(path.join(args.out, file), Buffer.from(res.bytes));
    return { url, file };
  } catch { return null; }
}

// ---- the capture, run in-page; returns the per-page record + hardening signals ----
// ---- dynamic-surface evidence (network side) — OPT-IN (`--dynamics`) --------
// Records WHAT the page fetched while rendering — never what it means. Cheap
// per-page REACH signals for the dynamics sub-skill: `dynamics-detect.mjs`
// probes archetypes in depth and folds these per-page sections (`--reach`) into
// each finding's reach. Migration-bound: prepare-migration, replica and migrate
// pass `--dynamics`; a bare extract, uplift and audit never do (dynamics is a
// migration concern, not a redesign one).
const DYNAMIC_MAX_ENDPOINTS = 150;
const DYNAMIC_MAX_HOSTS = 60;
const JSON_CT = /application\/(json|[a-z0-9.+-]*\+json)|text\/json|application\/graphql/i;

// collapse ids so /api/products/1234 and /api/products/5678 read as one endpoint
function pathPattern(u) {
  return u.pathname
    .replace(/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi, '{uuid}')
    .replace(/\/[0-9a-f]{16,}(?=\/|$)/gi, '/{hash}')
    .replace(/\/\d+(?=\/|$)/g, '/{n}');
}
function queryKeys(u) { return [...new Set([...u.searchParams.keys()])].sort(); }
// loose eTLD+1: enough to tell cdn.brand.com from analytics.vendor.com
function registrable(host) { return host.split('.').slice(-2).join('.'); }

function attachDynamicRecorder(page) {
  const endpoints = new Map(); // "METHOD host/path-pattern" → row
  const scriptHosts = new Map(); // host → count
  let truncated = false;
  page.on('response', (resp) => {
    try {
      const req = resp.request();
      const type = req.resourceType();
      const ct = (resp.headers()['content-type'] || '').split(';')[0].trim().toLowerCase();
      const u = new URL(resp.url());
      if (!/^https?:$/.test(u.protocol)) return;
      if (type === 'script') { scriptHosts.set(u.host, (scriptHosts.get(u.host) || 0) + 1); return; }
      if (type === 'document') return; // the page itself (and iframes' documents)
      const dataLike = type === 'xhr' || type === 'fetch' || type === 'eventsource' || JSON_CT.test(ct);
      if (!dataLike) return;
      const method = req.method();
      const key = `${method} ${u.host}${pathPattern(u)}`;
      const len = Number(resp.headers()['content-length']) || null;
      const row = endpoints.get(key);
      if (row) { row.hits += 1; if (len) row.bytes = Math.max(row.bytes || 0, len); return; }
      if (endpoints.size >= DYNAMIC_MAX_ENDPOINTS) { truncated = true; return; }
      endpoints.set(key, {
        method, host: u.host, path: pathPattern(u), query: queryKeys(u), resourceType: type,
        contentType: ct || null, status: resp.status(), bytes: len, hits: 1, example: `${u.origin}${u.pathname}`,
      });
    } catch { /* evidence only — never fail a capture on it */ }
  });
  return {
    finish(finalUrl) {
      const site = registrable(new URL(finalUrl).host);
      return {
        endpoints: [...endpoints.values()].map((r) => ({ ...r, sameSite: registrable(r.host) === site })),
        thirdPartyScriptHosts: [...scriptHosts.entries()]
          .filter(([h]) => registrable(h) !== site)
          .sort((a, b) => b[1] - a[1]).slice(0, DYNAMIC_MAX_HOSTS)
          .map(([host, count]) => ({ host, count })),
        truncated,
      };
    },
  };
}

// site-level roll-up (written to _crawl-log.json#dynamicSurface): which
// endpoints / hosts / frameworks / form targets recur across pages, with up to
// three example slugs each — the view Phase 4.5 reads first.
function newDynamicRollup() {
  return { endpoints: new Map(), thirdPartyScriptHosts: new Map(), frameworkHints: new Map(), globalState: new Map(), formTargets: new Map(), pages: 0, pagesWithSameSiteData: 0, pagesWithSearchForm: 0, pagesHydrated: 0, truncatedPages: 0 };
}
function bump(map, key, slug, extra) {
  const row = map.get(key) || { ...extra, pages: 0, examples: [] };
  row.pages += 1;
  if (row.examples.length < 3) row.examples.push(slug);
  map.set(key, row);
}
function rollupDynamic(acc, d, slug) {
  acc.pages += 1;
  if (d.truncated) acc.truncatedPages += 1;
  if (d.summary.sameSiteEndpoints) acc.pagesWithSameSiteData += 1;
  if (d.summary.searchForms) acc.pagesWithSearchForm += 1;
  if (d.summary.hydrated) acc.pagesHydrated += 1;
  for (const e of d.endpoints) bump(acc.endpoints, `${e.method} ${e.host}${e.path}`, slug, { method: e.method, host: e.host, path: e.path, query: e.query, resourceType: e.resourceType, contentType: e.contentType, sameSite: e.sameSite, example: e.example });
  for (const h of d.thirdPartyScriptHosts) bump(acc.thirdPartyScriptHosts, h.host, slug, { host: h.host });
  for (const f of d.frameworkHints) bump(acc.frameworkHints, f, slug, { hint: f });
  for (const g of d.globalState) bump(acc.globalState, g, slug, { name: g });
  for (const f of d.forms) bump(acc.formTargets, `${f.method} ${f.action || '(js-handled)'}`, slug, { action: f.action, method: f.method, sameOrigin: f.sameOrigin, search: f.search, fieldNames: f.fieldNames });
}
function finalizeDynamic(acc) {
  const list = (m, cap) => [...m.values()].sort((a, b) => b.pages - a.pages).slice(0, cap);
  return {
    pages: acc.pages,
    pagesWithSameSiteData: acc.pagesWithSameSiteData,
    pagesWithSearchForm: acc.pagesWithSearchForm,
    pagesHydrated: acc.pagesHydrated,
    truncatedPages: acc.truncatedPages,
    endpoints: list(acc.endpoints, 300),
    thirdPartyScriptHosts: list(acc.thirdPartyScriptHosts, DYNAMIC_MAX_HOSTS),
    frameworkHints: list(acc.frameworkHints, 20),
    globalState: list(acc.globalState, 20),
    formTargets: list(acc.formTargets, 50),
  };
}

function capture() {
  const vis = (el) => {
    if (!el || el.nodeType !== 1) return false;
    if (el.closest('[aria-hidden="true"],[hidden]')) return false;
    const cs = getComputedStyle(el);
    if (cs.display === 'none' || cs.visibility === 'hidden' || +cs.opacity === 0) return false;
    const r = el.getBoundingClientRect();
    if (r.width < 2 || r.height < 2) return false; // zero-area
    if (r.bottom < -2000 || r.right < -2000) return false; // far off-screen
    return true;
  };
  const INTERSTITIAL = /(temporarily unavailable|page unavailable|continuing to a page|go back to spanish|continue in english|this site uses cookies|accept all cookies|change cookie settings|privacy notice)/i;
  const isInterstitial = (t) => t && INTERSTITIAL.test(t.trim());

  let filtered = 0;
  const text = (el) => (el.textContent || '').replace(/\s+/g, ' ').trim();

  const meta = (n) => document.querySelector(`meta[name="${n}"]`)?.content
    || document.querySelector(`meta[property="${n}"]`)?.content || null;

  // headings: visible only, drop interstitial copy
  const headings = [...document.querySelectorAll('h1,h2,h3,h4,h5,h6')].filter((h) => {
    if (!vis(h)) return false;
    if (isInterstitial(text(h))) { filtered += 1; return false; }
    return true;
  }).map((h) => ({ tag: h.tagName.toLowerCase(), text: text(h) })).filter((h) => h.text);

  const main = document.querySelector('main') || document.body;
  // body paragraphs: visible, non-interstitial
  const body = [...main.querySelectorAll('p,blockquote,li')].filter((p) => {
    if (!vis(p)) return false;
    const t = text(p);
    if (!t || t.length < 2) return false;
    if (isInterstitial(t)) { filtered += 1; return false; }
    return true;
  }).map(text);

  // CTAs (visible button-like)
  const ctas = [...document.querySelectorAll('a[href],button,[role="button"]')].filter(vis)
    .map((a) => ({ label: text(a), href: a.getAttribute('href') || null }))
    .filter((c) => c.label && !isInterstitial(c.label)).slice(0, 100);

  // links
  const links = [...new Set([...document.querySelectorAll('a[href]')].map((a) => a.href))];

  // media — tracking pixels (lone off-origin <=2px) do NOT count as media
  const imgs = [...document.querySelectorAll('img')].map((im) => ({
    src: im.currentSrc || im.src, alt: im.alt || '', w: im.naturalWidth, h: im.naturalHeight,
  }));
  const realImgs = imgs.filter((im) => im.src && im.w > 2 && im.h > 2
    && !/(^data:|1x1|pixel|track|beacon|\/p\?|\/b\?)/i.test(im.src));
  const cssBgs = [];
  for (const el of document.querySelectorAll('*')) {
    const bg = getComputedStyle(el).backgroundImage;
    if (bg && bg !== 'none' && /url\(/.test(bg)) {
      const r = el.getBoundingClientRect();
      const m = bg.match(/url\(["']?([^"')]+)/);
      if (r.width >= 100 && r.height >= 80 && m) cssBgs.push(m[1]);
    }
  }

  // MODAL / AJAX detail: read textContent of dialog/modal containers EVEN IF hidden
  // (XHR-populated detail sits in a display:none .modal until opened).
  const modals = [...document.querySelectorAll('[role="dialog"],[aria-modal="true"],.modal,.modal-content')]
    .map((m) => text(m)).filter((t) => t && t.length > 40).slice(0, 10);

  const mainText = text(main);
  // custom props — discovery-vs-value split:
  //   * the stylesheet walk DISCOVERS property NAMES declared on :root/html-ish
  //     selectors, recursing into @media/@supports groups AND @import'ed sheets
  //     (a CSSImportRule exposes .styleSheet, not .cssRules — WordPress/legacy
  //     CMS token sheets commonly arrive via @import);
  //   * the recorded VALUE is always the LIVE one from
  //     getComputedStyle(documentElement). A declared value is accepted as
  //     fallback ONLY from unconditional rules (not inside any grouping rule
  //     with a condition, nor a conditional @import/link media) whose selector
  //     list contains exactly ':root' or 'html'. Names that only appear in
  //     conditional/themed rules (e.g. `:root.dark`, `@media (…)`) and compute
  //     empty are skipped — the rendered page never used them.
  const propNames = new Set();
  const declaredFallback = {};
  const isConditionalMedia = (media) => !!(media && media.mediaText && !/^(all)?$/i.test(media.mediaText.trim()));
  const walkRules = (rules, conditional) => {
    for (const rule of rules || []) {
      if (rule.type === 3 /* CSSRule.IMPORT_RULE */ || (typeof CSSImportRule !== 'undefined' && rule instanceof CSSImportRule)) {
        try {
          if (rule.styleSheet) walkRules(rule.styleSheet.cssRules, conditional || isConditionalMedia(rule.media));
        } catch { /* cross-origin imported sheet */ }
        continue;
      }
      if (rule.style && rule.selectorText) {
        const selectors = rule.selectorText.split(',').map((s) => s.trim());
        if (selectors.some((s) => /^(:root|html)\b/.test(s))) {
          const unconditionalRoot = !conditional && selectors.some((s) => s === ':root' || s === 'html');
          for (const p of rule.style) {
            if (!p.startsWith('--')) continue;
            propNames.add(p);
            // last unconditional exact-:root/html declaration wins (cascade order)
            if (unconditionalRoot) declaredFallback[p] = rule.style.getPropertyValue(p).trim();
          }
        }
      }
      if (rule.cssRules && rule.cssRules.length) {
        // grouping rule: @media/@supports carry a condition; @layer etc. do not
        const groupConditional = conditional || typeof rule.conditionText === 'string';
        try { walkRules(rule.cssRules, groupConditional); } catch { /* skip */ }
      }
    }
  };
  for (const sheet of document.styleSheets) {
    try { walkRules(sheet.cssRules, isConditionalMedia(sheet.media)); } catch { /* cross-origin sheet */ }
  }
  for (const p of document.documentElement.style) {
    if (p.startsWith('--')) {
      propNames.add(p);
      declaredFallback[p] = document.documentElement.style.getPropertyValue(p).trim();
    }
  }
  const rootStyle = getComputedStyle(document.documentElement);
  const customProps = {};
  for (const name of propNames) {
    const live = rootStyle.getPropertyValue(name).trim();
    if (live) customProps[name] = live;
    else if (declaredFallback[name]) customProps[name] = declaredFallback[name];
    // else: conditional/themed-only name with empty computed value — skip
  }

  // substance / SPA-shell signal
  const distinctHeadings = new Set(headings.map((h) => h.text)).size;
  const spaShellSuspect = distinctHeadings < 2 && mainText.length < 200 && realImgs.length === 0;

  // content hash for cross-page duplicate detection (detail == listing)
  const contentHash = `${headings.map((h) => h.text).join('|')}::${mainText.slice(0, 4000)}`;

  // code blocks: pre/code contents verbatim (stardust-style e2e finding — on a
  // developer-tool site the install commands are the most load-bearing content
  // and innerText body capture skips them). Visible pres only; innerText keeps
  // line structure.
  const codeBlocks = [...document.querySelectorAll('pre')].filter(vis)
    .map((el) => (el.innerText || '').trim()).filter(Boolean);

  // dynamic-surface evidence (DOM side): data blobs, hydration hints, forms.
  // Evidence only — prepare-migration Phase 4.5 classifies; merged with the
  // network-side recorder into `dynamic` by capturePage.
  const inlineData = [...document.querySelectorAll('script[type^="application/"][type*="json"],script#__NEXT_DATA__')]
    .filter((s) => !/ld\+json/i.test(s.type || ''))
    .slice(0, 20)
    .map((s) => {
      let keys = null;
      try { const j = JSON.parse(s.textContent); keys = j && typeof j === 'object' ? Object.keys(j).slice(0, 12) : null; } catch { /* not parseable */ }
      return { id: s.id || null, type: s.type || null, bytes: (s.textContent || '').length, topLevelKeys: keys };
    });
  const globalState = ['__NEXT_DATA__', '__NUXT__', '__INITIAL_STATE__', '__PRELOADED_STATE__', '__APOLLO_STATE__', '__remixContext', '__SVELTEKIT__', 'drupalSettings', 'wpApiSettings', 'Shopify', 'dataLayer']
    .filter((k) => { try { return k in window; } catch { return false; } });
  const q = (sel) => { try { return !!document.querySelector(sel); } catch { return false; } };
  const frameworkHints = [
    q('#__next') && 'next',
    q('#___gatsby') && 'gatsby',
    q('#__nuxt,#__layout') && 'nuxt',
    q('[data-reactroot],[data-reactid]') && 'react',
    q('[ng-version]') && 'angular',
    q('[data-v-app],[data-server-rendered]') && 'vue',
    q('[data-sveltekit-preload-data],[data-sveltekit-hydrate]') && 'sveltekit',
    q('astro-island') && 'astro',
    q('[data-turbo],[data-turbolinks]') && 'turbo',
    q('[data-wf-page],[data-wf-site]') && 'webflow',
    q('link[href*="/wp-content/"],script[src*="/wp-content/"]') && 'wordpress',
    q('script[src*="cdn.shopify.com"]') && 'shopify',
    q('.hs-form,[data-hs-forms-root],script[src*="hsforms"]') && 'hubspot-forms',
    q('script[src*="marketo"],form[id^="mktoForm"]') && 'marketo-forms',
    q('.aem-Grid,[data-cmp-is]') && 'aem-sites',
  ].filter(Boolean);
  const forms = [...document.querySelectorAll('form')].filter(vis).slice(0, 20).map((f) => {
    const inputs = [...f.querySelectorAll('input,select,textarea')].filter((i) => !['hidden', 'submit', 'button', 'reset'].includes((i.type || '').toLowerCase()));
    const names = inputs.map((i) => i.name || i.id || '').filter(Boolean);
    const rawAction = f.getAttribute('action');
    let action = null;
    try { action = new URL(rawAction || location.href, location.href); } catch { /* keep null */ }
    const search = f.getAttribute('role') === 'search'
      || inputs.some((i) => (i.type || '').toLowerCase() === 'search')
      || names.some((n) => /^(q|s|query|search|keyword|keywords|term)$/i.test(n))
      || (!!action && /search/i.test(action.pathname));
    return {
      action: action ? `${action.origin}${action.pathname}` : null,
      hasAction: !!rawAction, // no action attribute → almost always JS-submitted
      method: (f.getAttribute('method') || 'get').toLowerCase(),
      sameOrigin: action ? action.origin === location.origin : true,
      fieldCount: inputs.length,
      fieldNames: [...new Set(names)].slice(0, 12),
      search,
    };
  });
  const ariaLiveRegions = document.querySelectorAll('[aria-live]:not([aria-live="off"])').length;
  // reach signals for dynamics-detect --reach: modal-trigger markers and player ids per page
  const triggers = [...document.querySelectorAll('a, button')].map((el) => {
    const cls = el.getAttribute('class') || ''; const attrs = [...el.attributes].map((a) => a.name);
    const marker = (cls.match(/[\w-]*(modal|dialog|lightbox|popup)[\w-]*/i) || [])[0] || attrs.find((n) => /modal|dialog|lightbox|popup/i.test(n)) || (el.getAttribute('aria-haspopup') === 'dialog' ? 'aria-haspopup=dialog' : null);
    return marker && !/close|dismiss/i.test(cls) ? { marker, href: el.getAttribute('href') || null } : null;
  }).filter(Boolean).slice(0, 40);
  const mediaIds = [...document.querySelectorAll('video-js, [data-video-id], [data-videoid], iframe[src*="player" i]')].map((el) => el.getAttribute('data-video-id') || el.getAttribute('data-videoid') || el.getAttribute('src')).filter(Boolean).slice(0, 20);

  return {
    dynamicDom: { inlineData, globalState, frameworkHints, forms, ariaLiveRegions, triggers, mediaIds },
    finalUrl: location.href,
    title: document.title || null,
    description: meta('description'),
    og: { title: meta('og:title'), description: meta('og:description'), image: meta('og:image'), type: meta('og:type') },
    headings,
    body,
    codeBlocks,
    ctas,
    links,
    media: {
      imgs: realImgs,
      allImgCount: imgs.length,
      cssBackgrounds: [...new Set(cssBgs)],
      modals,
      videos: [...document.querySelectorAll('video')].filter(vis).map((v) => ({
        src: v.currentSrc || v.src || v.querySelector('source')?.src || null,
        poster: v.poster || null,
        autoplay: v.autoplay,
        loop: v.loop,
        muted: v.muted,
      })),
      iframes: [...document.querySelectorAll('iframe')].filter(vis).map((f) => ({
        src: f.src || null,
        title: f.title || null,
      })),
    },
    customProps,
    _signals: {
      filteredInterstitials: filtered,
      distinctHeadings,
      mainTextLen: mainText.length,
      realImageCount: realImgs.length,
      trackingOnlyMedia: imgs.length > 0 && realImgs.length === 0,
      spaShellSuspect,
    },
    _contentHash: contentHash,
  };
}

/**
 * The page's screenshot for the vision gate (#602) → { mode, pageSize, layoutOverflow, reason }. The page is measured
 * first (live-session pageSizeInPage / shotPlan): a page at Chrome's layout maximum — a CSS defect on the source — or
 * wider than 4 viewports gets the viewport only, a page taller than SHOT_MAX_HEIGHT is captured down to it ('capped'),
 * any other page in full. A full-page capture that throws is retried viewport-only on the same page — unless the
 * renderer crashed: then on a fresh page from `reopen` (null = none), closed afterwards, never on the dead one.
 * mode: 'fullPage' | 'capped' | 'viewport' | 'failed'; `reason` says why a capture is less than the full page.
 */
export async function screenshotPage(page, shotPath, { reopen = null } = {}) {
  const { pageSizeInPage, shotPlan, layoutOverflow } = await liveSession();
  let crashed = false;
  if (typeof page.on === 'function') page.on('crash', () => { crashed = true; });
  const viewport = (typeof page.viewportSize === 'function' && page.viewportSize()) || CRAWL_CONTEXT.viewport;
  const measured = await page.evaluate(pageSizeInPage).catch(() => null);
  const pageSize = measured && Number.isFinite(measured.w) && Number.isFinite(measured.h) ? { w: measured.w, h: measured.h } : null;
  const plan = shotPlan(pageSize, viewport.width);
  const out = { mode: plan.mode, pageSize, layoutOverflow: Boolean(layoutOverflow(pageSize)), reason: plan.reason };
  const viewportShot = async (p) => p.screenshot({ path: shotPath, fullPage: false, timeout: 30000 });
  if (plan.mode === 'viewport') {
    try { await viewportShot(page); } catch (e) { return { ...out, mode: 'failed', reason: `${plan.reason}; the viewport screenshot failed too (${String(e.message || e).split('\n')[0]})` }; }
    return out;
  }
  try {
    await page.screenshot({ path: shotPath, fullPage: true, timeout: 30000, ...(plan.clip ? { clip: plan.clip } : {}) });
    return out;
  } catch (e) {
    const why = String(e.message || e).split('\n')[0];
    let target = page;
    if (crashed || (typeof page.isClosed === 'function' && page.isClosed())) {
      if (!reopen) return { ...out, mode: 'failed', reason: `the renderer crashed during the full-page screenshot (${why})` };
      try { target = await reopen(); } catch (re) { return { ...out, mode: 'failed', reason: `the renderer crashed during the full-page screenshot (${why}); reopening the page failed (${String(re.message || re).split('\n')[0]})` }; }
    }
    try {
      await viewportShot(target);
      return { ...out, mode: 'viewport', reason: `the full-page screenshot failed (${why})${target === page ? '' : ' and crashed the renderer: the viewport came from a fresh page'}` };
    } catch (ve) {
      return { ...out, mode: 'failed', reason: `the full-page screenshot failed (${why}); the viewport screenshot too (${String(ve.message || ve).split('\n')[0]})` };
    } finally {
      if (target !== page) await target.close().catch(() => {});
    }
  }
}

/**
 * Capture one page on a fresh tab of `context` under the per-page deadline (args.pageTimeoutMs; 0 = none): a page
 * that never settles is recorded as TimeoutError instead of eating the run's cap. The deadline closes the tab, so
 * the abandoned navigation or probe dies with it. Every exit path releases the page. `sharedBindings` (tests) stands in
 * for the front door's live-session hooks: they pace, record and persist for this capture.
 */
export async function capturePage(context, url, slug, args, sharedBindings = null) {
  const page = await context.newPage();
  try {
    return await withPageDeadline(captureOnPage(page, url, slug, args, sharedBindings, context), args.pageTimeoutMs);
  } finally {
    // every exit path — success, validation throw, goto error, the deadline — releases the
    // page, or failure-heavy crawls accumulate open tabs in the long-lived worker context.
    await page.close().catch(() => {});
  }
}
async function captureOnPage(page, url, slug, args, sharedBindings = null, context = null) {
  const live = sharedBindings ? crawlerLiveBindings(sharedBindings) : null;
  const navigateLive = live ? live.runLiveNavigation : sharedNavigate;
  const recordSuccess = live ? live.recordLiveNavigationSuccess : recordSharedSuccess;
  const persistState = live ? live.persistLiveStorageState : persistSharedState;
  const recordBotSignal = live ? live.recordLiveBotSignal : recordSharedBotSignal;
  const recorder = args.dynamics ? attachDynamicRecorder(page) : null; // opt-in; must precede goto — load-time fetches are the evidence
  const { goto } = await liveSession();
  let resp = await navigateLive(page, url, () => goto(page, url, { waitUntil: 'domcontentloaded', timeout: 45000 }));
  // response validation
  if (!resp) throw Object.assign(new Error('no response'), { errorClass: 'TimeoutError' });
  // bot-management challenge (Cloudflare cf-mitigated: challenge, etc.): the
  // worker's fresh context is re-challenged even after the probe cleared it, so
  // give the interstitial its JS-solve window and reload before validating —
  // a challenge that does not clear is recorded as BotChallenge naming the
  // signature, never as a bare HTTPError (a 403 under cf-mitigated on every inner
  // page after a clean entry — recorded on a Cloudflare-managed site).
  resp = await clearChallenge(page, resp, {
    recordBotSignal, navigate: navigateLive, signalRecorded: live && live.liveResponseSignalRecorded,
  });
  const edge = challengeMarker(resp);
  if (edge) throw Object.assign(new Error(`bot-management challenge not cleared: ${edge}`), { errorClass: 'BotChallenge' });
  let status = resp.status();
  // 404 on a slash variant: retry ONCE with the trailing slash flipped before
  // recording a failure (stardust-style e2e finding — slash-required hosts).
  let resolvedUrl = url;
  if (status === 404) {
    const u = new URL(url);
    if (u.pathname.length > 1) {
      u.pathname = u.pathname.endsWith('/') ? u.pathname.replace(/\/+$/, '') : `${u.pathname}/`;
      // guarded + short timeout: a hanging flipped-variant probe must not
      // replace the crisp HTTPError 404 with a raw TimeoutError.
      const retry = await navigateLive(page, u.href, () => goto(page, u.href, { waitUntil: 'domcontentloaded', timeout: 15000 }))
        .catch(() => null);
      if (retry && retry.status() < 400) {
        console.error(`[crawl] slash-retry OK ${url} -> ${u.href}`);
        resp = retry; status = retry.status(); resolvedUrl = u.href;
      }
    }
  }
  if (status >= 400) throw Object.assign(new Error(`HTTP ${status}`), { errorClass: 'HTTPError' });
  const ct = resp.headers()['content-type'] || '';
  if (!/text\/html|application\/xhtml/.test(ct)) throw Object.assign(new Error(`content-type ${ct}`), { errorClass: 'ContentTypeError' });

  if (args.consent) await dismissConsent(page);
  await recordSuccess(resolvedUrl).catch(() => {});
  await persistState(page, resolvedUrl).catch(() => {});
  await page.waitForTimeout(WAIT_MS[args.wait] || WAIT_MS.medium);
  // a 2xx verification interstitial served as a normal page ("One moment, please…", "Just a moment…", CloudFront's
  // "Request blocked" body): the headers say nothing, the page is read (pageWall) — recorded as BotChallenge, never
  // captured as content.
  const wall = await pageWall(page);
  if (wall.skipped) console.error(`[crawl] wall check skipped on ${slug}: ${wall.skipped}`);
  if (wall.marker) throw Object.assign(new Error(`bot-management interstitial served as HTTP ${status}: ${wall.marker}`), { errorClass: 'BotChallenge' });
  // 4-step scroll to trigger lazy content
  for (let y = 0; y <= 1; y += 0.34) {
    await page.evaluate((f) => window.scrollTo(0, document.body.scrollHeight * f), y);
    await page.waitForTimeout(400);
  }
  await page.evaluate(() => window.scrollTo(0, 0));
  // settle after return-to-top: entry animations (hero reveals) must reach
  // their final state before the visibility filter reads computed opacity, or
  // the animated h1 is silently dropped. reducedMotion emulation neutralizes
  // most of it; the settle covers JS-driven reveals. Deliberately a FLAT wait:
  // gating on document.getAnimations() was tried and re-dropped the h1 — a
  // JS-delayed reveal has no running animation at check time, so the gate
  // resolves before the reveal even starts. The 800ms floor is load-bearing.
  await page.waitForTimeout(800);

  const rec = await page.evaluate(capture);
  // dynamic surface (opt-in): network recorder + DOM-side evidence → one `dynamic` section
  if (!recorder) delete rec.dynamicDom;
  else {
    const net = recorder.finish(rec.finalUrl || resolvedUrl);
    const dom = rec.dynamicDom;
    delete rec.dynamicDom;
    rec.dynamic = {
      ...net,
      ...dom,
      summary: {
        sameSiteEndpoints: net.endpoints.filter((e) => e.sameSite).length,
        thirdPartyEndpoints: net.endpoints.filter((e) => !e.sameSite).length,
        thirdPartyScriptHosts: net.thirdPartyScriptHosts.length,
        inlineDataBlobs: dom.inlineData.length,
        forms: dom.forms.length,
        searchForms: dom.forms.filter((f) => f.search).length,
        hydrated: dom.frameworkHints.length > 0 || dom.globalState.some((g) => g !== 'dataLayer'),
      },
    };
  }
  // rendered DOM sidecar — the settled document as the instrument saw it
  // (written by the caller as pages/<slug>.html; parse offline, never re-scrape).
  rec._renderedHtml = await page.content();
  // soft-404: empty page (no text, no headings, no media, no forms)
  if (!rec.headings.length && rec._signals.mainTextLen === 0 && rec._signals.realImageCount === 0) {
    throw Object.assign(new Error('empty page — possibly soft-404'), { errorClass: 'EmptyPageError' });
  }
  // full-page screenshot for the Phase 2.5 vision gate — measured first, captured by screenshotPage.
  const shotsDir = path.join(args.out, 'assets', 'screenshots');
  await mkdir(shotsDir, { recursive: true });
  const shotPath = path.join(shotsDir, `${slug}.png`);
  const shot = await screenshotPage(page, shotPath, {
    reopen: context && (async () => {
      const fresh = await context.newPage();
      await navigateLive(fresh, resolvedUrl, () => goto(fresh, resolvedUrl, { waitUntil: 'domcontentloaded', timeout: 45000 }));
      await fresh.waitForTimeout(WAIT_MS[args.wait] || WAIT_MS.medium);
      return fresh;
    }),
  });
  const screenshotMode = shot.mode;
  rec.screenshot = screenshotMode === 'failed' ? null : `assets/screenshots/${slug}.png`;
  rec._signals.screenshotMode = screenshotMode;
  if (shot.pageSize) rec._signals.pageSize = shot.pageSize;
  if (shot.layoutOverflow) rec._signals.layoutOverflow = true;
  if (shot.reason) rec._signals.screenshotReason = shot.reason;
  // live-render evidence per SKILL.md § Phase 2 / current-state-schema.md —
  // validateProvenance() downstream refuses pages without these five fields.
  if (resolvedUrl !== url) rec._resolvedUrl = resolvedUrl;
  rec._provenance = {
    renderedBy: 'playwright',
    fetchedAt: new Date().toISOString(),
    waitMode: args.wait || 'medium',
    waitMs: WAIT_MS[args.wait] || WAIT_MS.medium,
    httpStatus: status,
  };
  return rec;
}

async function main() {
  const args = parseArgs(process.argv);
  const outPages = path.join(args.out, 'pages');
  await mkdir(outPages, { recursive: true });

  let stealth = false;
  let { browser, technique } = await launchWithFallback();
  let context = await newContext(browser, stealth, args.url);
  let probe = await context.newPage();

  // bot-management probe on the entry URL (probeEntry). Three block modes, all named within seconds:
  //   1. a network fingerprint block — the goto THROWS (isFingerprintBlock);
  //   2. a challenge / edge block — the goto SUCCEEDS but the response is a 403/429/503 interstitial with an edge
  //      signature (challengeMarker; CloudFront / AWS WAF included) — this one once slipped through the probe and
  //      only failed at capture-time (Cloudflare-fronted site finding);
  //   3. a 2xx VERIFICATION INTERSTITIAL the headers say nothing about ("One moment, please…" reloading itself) —
  //      read from the page; a crawler that trusted the 200 waited out the run's whole cap on it.
  // Any of them → ONE relaunch (launchEscalated: fresh real Chrome, window-free, the existing launch flags unless
  // STARDUST_STEALTH=0) + the challenge-solve window, then fail fast naming the wall — never capture it as content
  // (stop-condition (a) for the skill).
  const botBlock = await probeEntry(probe, args.url, { timeout: 30000 }); // { kind, marker } | null — the first verdict, for the log
  if (botBlock) {
    console.error(`[crawl] bot-management block on the entry page (${botBlock.kind}: ${botBlock.marker}) — relaunching real Chrome (channel: chrome, window-free${stealthEnabled() ? '' : ', STARDUST_STEALTH=0: automation-signal handling off'}) and running the challenge-solve window`);
    const escalated = await launchEscalated();
    if (escalated) {
      await browser.close().catch(() => {});
      browser = escalated;
      technique = 'chrome-escalated';
      stealth = stealthEnabled();
    } else {
      await context.close().catch(() => {});
    }
    context = await newContext(browser, stealth, args.url);
    probe = await context.newPage();
    const still = await probeEntry(probe, args.url, { timeout: 45000, solve: true });
    if (still) {
      await browser.close().catch(() => {});
      throw Object.assign(
        new Error(`bot-management ${still.kind} not cleared after the real-Chrome relaunch + challenge-solve window: ${still.marker} — the site requires an interactive challenge solve; complete it by hand in a session with a window (STARDUST_HEADED_WINDOW=1 on the gate instruments) or capture it another way`),
        { errorClass: 'BotChallengeError' },
      );
    }
  }
  await persistSharedState(probe).catch(() => {});

  // adopt the post-redirect origin (apex→www etc.): the same-origin filter and
  // sitemap fetch must use where the site actually lives, or discovery silently
  // collapses to 1 page (agency-site e2e finding).
  let originRedirect = null;
  try {
    const landed = new URL(probe.url());
    if (landed.origin !== args.origin) {
      originRedirect = { from: args.origin, to: landed.origin };
      console.error(`[crawl] origin redirect ${args.origin} -> ${landed.origin} — adopting post-redirect origin`);
      args.origin = landed.origin;
      args.url = landed.href;
    }
  } catch { /* keep declared origin */ }

  const urls = await discover(args, probe);
  // favicon rides the probe page (already on the entry URL) — runs in every
  // mode, so bounded extracts can't silently drop it (CEN-4).
  const favicon = await captureFavicon(probe, args);
  if (favicon) console.error(`[crawl] favicon captured: ${favicon.file} (${favicon.url})`);
  else console.error('[crawl] WARN no favicon captured — no link[rel~=icon] and /favicon.ico unreachable; deploy will ship the default icon unless one is provided');
  await probe.close();
  console.error(`[crawl] technique=${technique} pages=${urls.length}`);

  const log = { discovery: { fetchTechnique: technique, count: urls.length, concurrency: args.concurrency, ...(botBlock ? { botBlock: botBlock.kind, botMarker: botBlock.marker } : {}), ...(originRedirect ? { originRedirect } : {}) }, consent: { method: args.consent ? 'auto' : 'skipped' }, favicon: favicon || null, crawl: { failures: [], degraded: [], pageTimeoutMs: args.pageTimeoutMs } };
  let ok = 0;
  await context.close();

  // worker pool: N parallel BrowserContexts drain the shared queue. Consent is
  // re-established per page (dismissConsent runs inside capturePage), so each
  // fresh context is covered without cross-context cookie sharing.
  // During capture we only RECORD content hashes (indexed by queue position);
  // duplicate attribution happens in a deterministic post-pass below.
  const results = new Array(urls.length).fill(null); // { slug, file, hash } per queue index
  const dynamicRollup = newDynamicRollup();
  const slugs = assignSlugs(urls);
  let nextIdx = 0;
  async function worker() {
    const ctx = await newContext(browser, stealth, args.url);
    while (nextIdx < urls.length) {
      const idx = nextIdx;
      nextIdx += 1;
      const url = urls[idx];
      const slug = slugs[idx];
      try {
        const rec = await capturePage(ctx, url, slug, args);
        if (rec.dynamic) rollupDynamic(dynamicRollup, rec.dynamic, slug);
        const hash = crypto.createHash('sha1').update(rec._contentHash).digest('hex');
        delete rec._contentHash;
        // slash-retry rescue: record the URL that actually served the page and
        // an audit-trail entry — downstream consumers of `url` must not re-hit
        // the 404 variant the crawler already learned to avoid.
        const recordUrl = rec._resolvedUrl || url;
        if (rec._resolvedUrl) {
          log.crawl.slashRetries = log.crawl.slashRetries || [];
          log.crawl.slashRetries.push({ requested: url, resolved: rec._resolvedUrl, slug });
          delete rec._resolvedUrl;
        }
        const file = path.join(outPages, `${slug}.json`);
        const htmlFile = path.join(outPages, `${slug}.html`);
        await writeFile(htmlFile, rec._renderedHtml);
        delete rec._renderedHtml;
        rec.renderedHtml = `pages/${slug}.html`;
        const { _provenance, ...rest } = rec;
        // top-level renderedBy/fetchedAt are legacy-reader aliases of the same
        // _provenance fields — _provenance is the authoritative contract.
        await writeFile(file, JSON.stringify({ _provenance, slug, url: recordUrl, renderedBy: _provenance.renderedBy, fetchedAt: _provenance.fetchedAt, ...rest }, null, 2));
        results[idx] = { slug, file, hash, url: recordUrl, finalUrl: rec.finalUrl, links: rec.links }; // url/finalUrl/links feed the capture-gap post-pass
        ok += 1;
        const s = rec._signals;
        const dy = rec.dynamic?.summary || {};
        const warn = [s.spaShellSuspect && 'SPA-SHELL?', s.trackingOnlyMedia && 'TRACKING-PIXEL-ONLY', s.filteredInterstitials && `filtered:${s.filteredInterstitials}`, dy.sameSiteEndpoints && `data-endpoints:${dy.sameSiteEndpoints}`, dy.searchForms && 'SEARCH-FORM', dy.hydrated && 'HYDRATED'].filter(Boolean).join(' ');
        // a page without a usable screenshot, or one whose own CSS overflowed, is a WARN with the reason (#602):
        // `screenshotReason` says whether a recrawl can help — never for a layout overflow
        if (s.screenshotMode === 'failed' || s.layoutOverflow) {
          log.crawl.degraded.push({ url: recordUrl, slug, screenshotMode: s.screenshotMode, ...(s.layoutOverflow ? { layoutOverflow: true } : {}), reason: s.screenshotReason || null });
          console.error(`[crawl] WARN ${slug}  screenshot ${s.screenshotMode}: ${s.screenshotReason || 'no reason recorded'}  ${warn}`);
        } else console.error(`[crawl] OK   ${slug}  ${[s.screenshotMode === 'fullPage' ? '' : `shot:${s.screenshotMode}`, warn].filter(Boolean).join(' ')}`);
      } catch (err) {
        log.crawl.failures.push({ url, slug, errorClass: err.errorClass || 'Error', message: String(err.message || err), at: new Date().toISOString() });
        console.error(`[crawl] FAIL ${slug}  ${err.errorClass || 'Error'}: ${err.message}`);
      }
    }
    await ctx.close();
  }
  await Promise.all(Array.from({ length: Math.min(args.concurrency, urls.length) }, worker));
  await browser.close();

  // cross-page duplicate (detail == listing) detection — deterministic post-pass
  // in original queue order: canonical = earliest-QUEUED page per content hash
  // (not whichever finished first under the pool); later ones marked duplicateOf.
  const canonicalByHash = new Map();
  for (const r of results) {
    if (!r) continue;
    if (!canonicalByHash.has(r.hash)) { canonicalByHash.set(r.hash, r.slug); continue; }
    const canonical = canonicalByHash.get(r.hash);
    const rec = JSON.parse(await readFile(r.file, 'utf8'));
    rec._signals = rec._signals || {};
    rec._signals.duplicateOf = canonical;
    await writeFile(r.file, JSON.stringify(rec, null, 2));
    console.error(`[crawl] DUP  ${r.slug}  DUP-OF:${canonical}`);
  }
  // locale-root capture gaps: the subtree a captured locale root links to but discovery never listed —
  // surfaced here, and as a ready --detail string for the extract `end` ledger line, not six hours later
  // in the rollout link audit.
  log.captureGaps = captureGaps(results.filter(Boolean));
  if (log.captureGaps.roots.length) {
    console.error(`[crawl] capture gaps: ${log.captureGaps.roots.map((r) => `${r.root} ${r.uncaptured.length} of ${r.linked} first-level target(s) uncaptured`).join('; ')} — list in _crawl-log.json#captureGaps; ledger --detail "${log.captureGaps.detail}"`);
  }
  if (args.dynamics) {
    log.dynamicSurface = finalizeDynamic(dynamicRollup);
    console.error(`[crawl] dynamic surface (reach): ${log.dynamicSurface.endpoints.filter((e) => e.sameSite).length} same-site data endpoints, ${log.dynamicSurface.pagesWithSearchForm} pages with a search form, ${log.dynamicSurface.pagesHydrated} hydrated — depth + classification: stardust:dynamics`);
  }
  // merge into existing _crawl-log.json if present
  const logPath = path.join(args.out, '_crawl-log.json');
  const prev = existsSync(logPath) ? JSON.parse(await readFile(logPath, 'utf8')) : {};
  await writeFile(logPath, JSON.stringify({ ...prev, ...log }, null, 2));
  console.error(`[crawl] done. ${ok}/${urls.length} captured, ${log.crawl.failures.length} failed${log.crawl.degraded.length ? `, ${log.crawl.degraded.length} without a full screenshot (crawl.degraded)` : ''}. log: ${logPath}`);
}

if (IS_MAIN) main().catch((e) => { console.error(`[crawl] fatal: ${e.message}`); process.exit(2); });

skills

README.md

tile.json