Ultimate Browsing

Web access for the hosts the user's browser cannot open (WAF / 403 / Cloudflare / bot-scored pages) and for platforms with a native reader. Reading and interaction on a page the user's browser can open belong to the browser skill's attached engine, not here. Start at the cheapest tier that can reach the blocked host and climb only when it cannot:

Tier 1 — insane-search (headless extraction + WAF bypass) -> Tier 1.5 — agent-reach (platform-native APIs, esp. Chinese platforms) -> Tier 2 — a real browser through omowright from js eval: 2a the owned engine (a browser your code launches, CloakBrowser for stealth), 2b the attached engine (the user's own signed-in browser).

PHASE 0 — ROUTE FIRST (MANDATORY)

User request
  |
  +- a page the user's browser can open (read, click, screenshot, QA, login)  NOT this skill (browser skill, attached engine)
  +- simple search query ------------------------------ NOT this skill (use web search)
  |
  +- URL blocked / 403 / Cloudflare / WAF ------------- TIER 1  insane-search
  +- YouTube/Vimeo/TikTok subtitles or metadata ------- TIER 1  insane-search (yt-dlp)
  +- archive snapshot of a dead or changed page ------- TIER 1  insane-search (surrogates, provenance-tagged)
  |
  +- Chinese platform (xhs/douyin/weibo/bilibili/v2ex/wechat)  TIER 1.5 agent-reach
  +- podcast transcript ------------------------------- TIER 1.5 agent-reach
  +- Twitter feed / LinkedIn profile / GitHub via CLI - TIER 1.5 agent-reach
  |
  +- Tier 1/1.5 returned empty or partial ------------- TIER 2  2a owned engine (stealth)
  +- a blocked host that also needs a click, a scroll or a screenshot  TIER 2  2a owned engine
  +- the user's account on a host the owned engine cannot pass  TIER 2  2b attached engine (their browser)

Read the matching reference before acting: references/insane-search/README.md, references/agent-reach/README.md, or references/chrome-stealth.md.

Tier 1 — insane-search (headless extraction)

When: blocked-URL bypass, media metadata, archive surrogates — no browser UI needed. Why first: ~10x faster than a browser, no process spin-up; handles most "fetch this blocked page" requests via curl_cffi TLS impersonation, yt-dlp (1858 sites), official public APIs, mobile URL transforms, Phase-2.5 surrogate archives (Wayback / archive.today snapshots, provenance-tagged — see references/insane-search/cache-archive.md), a key-gated Jina Reader (JINA_API_KEY), and a Playwright real-Chrome fallback. The engine lives inside this skill at engine/ and is invoked as a module. Surrogate results are dated COPIES: a result whose provenance is snapshot must be reported with its snapshot_timestamp, never presented as the live page.

# Core command — auto-detects WAF, runs the full fetch grid (run from the skill dir).
# Keep -B: this skill can live inside a signed app bundle, and bytecode caches written into engine/ break its signature.
python3 -B -m engine "https://example.com/blocked-page"
#   add --selector "<CSS>" for positive-proof validation, --device auto|desktop|mobile,
#   --trace to inspect every attempt, --json for machine-readable output.

# YouTube subtitles / metadata (no browser):
yt-dlp --write-sub --write-auto-sub --sub-lang "en,ko" --skip-download -o "/tmp/%(id)s" "<URL>"

# Reddit / HN / Bluesky / arXiv etc. use official public endpoints — see the Phase 0 index in
# references/insane-search/README.md (Twitter syndication, Reddit .json, HN Firebase, ...).

The full engine harness (rules R1-R7, the Phase 0 official-API index, the no-site-name rule, and the references/insane-search/*.md deep-dives for TLS, Playwright routing, Naver, media, etc.) is in references/insane-search/README.md. Read it before tuning the engine or adding a WAF profile.

Escalate to Tier 1.5 or Tier 2 when

• The target is a Chinese / social platform with a native reader -> Tier 1.5.
• insane-search returns empty/partial, or the blocked page needs JS interaction, a screenshot, or media playback -> Tier 2.

Tier 1.5 — agent-reach (platform-native readers)

When: the target is a platform with a first-class API/CLI that beats generic fetching — especially Chinese platforms that stealth browsers still cannot reach cleanly. Several channels are zero-config (Douyin, V2EX, Reddit, RSS, YouTube); others need a one-time auth you supply via environment variables if you have access (JINA_API_KEY for Jina Reader — anonymous access is dead, see references/insane-search/jina.md; TWITTER_* for X; a transcription key for podcasts).

mcporter call 'douyin.parse_douyin_video_info(share_link: "<URL>")'   # douyin, zero-config
curl -s -H "Authorization: Bearer ${JINA_API_KEY}" "https://r.jina.ai/https://weibo.com/<uid>/<pid>"   # weibo via Jina (key required)
yt-dlp --dump-json "<bilibili-url>"                            # Bilibili (overseas: add --cookies-from-browser)
curl -s "https://www.v2ex.com/api/topics/hot.json"            # V2EX public API

Routing table, per-platform auth (set TWITTER_* env vars, gh auth login, a transcription key — only if you have access), rate-limit notes, and known version quirks are in references/agent-reach/README.md.

Tier 2 — a real browser (real interaction)

When: Tier 1/1.5 failed on a blocked host, or that host needs interaction (clicks, scroll, screenshots, video). A page the user's browser can open is the browser skill's job, attached engine first.

Both tiers are omowright, staged inside the browser skill and loaded from js eval:

const { loadOmowright } = await import("<browser-skill-root>/scripts/omowright.mjs")
const { omowright } = await loadOmowright()

Tier 2a — owned engine (default)

A browser your code launches with a task-owned profile. connectPipe opens no listening port; connectCloakProfile launches CloakBrowser with a pinned fingerprint seed and is the path for WAF, Cloudflare and bot-scored pages.

const browser = await omowright.connectPipe({ browserPath, browserArgs: ["--headless", `--user-data-dir=${profile}`], storageRoot: profile })
try {
  const page = await browser.newTab(url)
  const tree = omowright.compactSnapshot(await page.snapshot())   // the read; refs come from it
  const snoop = omowright.createNetworkSnoop(page)                  // read the API JSON instead of the DOM when there is one
  await page.locator("e3").click()
  await Bun.write(pngPath, await page.screenshot())
} finally {
  await browser.close()                                            // then rm -rf the profile
}

The rest of the surface (CUA coordinates, captcha solving, routes, traces, frames, human handoff) is in the browser skill's references/owned-engine/. A stealth binary is not proof of access: inspect the rendered result and report challenges that remain.

Tier 2b — attached engine (the user's account on a blocked host)

When the blocked flow needs the user's account, drive the browser they are already signed into instead of cloning their profile: connectBrowserSkill() → session.navigate → bskSnapshot(session) / session.observe() → session.click → session.stop(). NEVER launch against or clear cookies/cache/site data from the user's live profile, and never fall back to the owned engine for an authenticated criterion: if no extension is connected, run the browser skill's onboarding script and relay its one human step. The full loop is the browser skill.

Cookie login (cross-platform)

A login the user holds goes through Tier 2b, never through their cookies: the browser skill forbids copying a session out of the user's real browser profile (references/owned-engine/network.md, Cookies).

scripts/extract_cookies.py reads cookies from the default profile store of a local Chromium-family or Firefox-family browser and optionally injects them over CDP. It resolves browser profile paths and decrypts cookie values per-OS (macOS Keychain, Linux libsecret, Windows DPAPI). Because it always reads that default store, run it only where that browser profile is not the user's live profile (for example a dedicated automation account whose browser the task owns); on the user's own machine it would copy their session, so use Tier 2b instead.

# Extract cookies to a file:
mkdir -p ~/.local/state/omo-cookies
python3 -B scripts/extract_cookies.py --browser chrome --domain youtube.com --output ~/.local/state/omo-cookies/youtube.cookies.json
# Extract and inject over CDP (needs a browser started with --remote-debugging-port=9242; connectPipe opens no port):
python3 -B scripts/extract_cookies.py --browser chrome --domain youtube.com --inject --cdp 9242

For a Tier 2a owned page, load the exported file and call omowright.injectCookies(page, cookies) instead (see references/chrome-stealth.md). Cookie export files are written with owner-only 0600 permissions. Do not place live auth cookies in shared temp directories or commit them to a repo. Cookie injection sends values to CDP over stdin rather than argv. Cookies apply on next navigation — reload after injecting. Google services use fingerprint-bound tokens that may not transfer across browser profiles. Limits in references/chrome-stealth.md.

Reference docs

Environment variables

# agent-reach auth: set the channel-specific env vars from each tool's docs only if you have access
# insane-search: JINA_API_KEY only for the Jina Reader route (the engine skips that surrogate without it)
# insane-search Python deps: the install one-liner in references/insane-search/README.md

Anti-patterns

• Do NOT launch Chrome stealth for plain text extraction — use Tier 1.
• Use stealth plugins only in an explicitly installed script environment, not injected into WebView.
• Close every WebView/browser context when done and remove only task-owned profile clones.
• Do NOT inject cookies without reloading the page.
• Do NOT hardcode site domains/selectors into engine/** or waf_profiles.yaml — runtime hints only (see the no-site-name rule in the insane-search reference).