web-archive

URL in, single self-contained HTML out at <project_root>\__references\__archives.web\<domain>\<slug>.<date>.html. Every CSS/JS/image/font inlined as data: URIs — works offline, portable across machines, no broken refs. Metadata is embedded inside the HTML. A SQLite index next to the archives provides fast url → path lookup.

When to use

• /vault-ingest receives a URL → archive as backup alongside markdown conversion
• X post / Reddit thread / blog post / article you're citing — capture before it disappears
• User says: "save this", "archive", "snapshot", "back up this link"

Don't use for: JSON / REST APIs (use WebFetch); one-shot read of one page (use WebFetch); local files you already control.

Output layout

<project_root>/__references/__archives.web/
├── .archive.sqlite                                # index (+ future cookies/sessions)
├── x.com/
│   └── tom_doerr-status-2059312274989256935.2026-05-27.html
└── docs.python.org/
    └── 3-library-asyncio.2026-05-27.html

Filename: <path-slug>.<YYYY-MM-DD>.html. Same URL re-captured the same day with different content → <slug>.<YYYY-MM-DD>T<HHMMSS>Z.html (both kept). Identical content → cache hit, no new file.

Python API

from web_archiver import archive, lookup

result = archive(
    url="https://x.com/tom_doerr/status/2059312274989256935",
    project_root=None,        # auto-detected from cwd if omitted
    backend="auto",
    options={"timeout_sec": 60},
)
# result.archive_path (Path), .sha256, .backend, .status, .captured_at, .url, .fresh

existing = lookup(url="https://x.com/...")   # Optional[ArchiveResult]

Project-root resolution: explicit arg → WEB_ARCHIVE_PROJECT_ROOT env → cwd walk-up for .claude//.pi//__references/ marker → cwd.

CLI

web-archive archive --url <url> [--project-root <path>] [--backend NAME]
                    [--timeout-sec N] [--force] [--json]
web-archive lookup --url <url> [--project-root <path>] [--json]

Exit codes: 0 ok, 1 failed / not-found, 2 bad args.

Backend selection (backend="auto")

Only singlefile is implemented today. Pinned-but-unavailable backends raise; auto downgrades silently.

Install

pip install -e <path-to>/web-archiver
npm install -g single-file-cli           # required backend

Common mistakes

• Passing a YouTube / Reddit / Twitter URL with backend="singlefile" → useless "log in" capture. Use auto.
• Treating fresh=False as failure → it's a cache hit. Check status.
• Calling for JSON API URLs → wrong tool. Use WebFetch.

See __docs/output-contract.md for the full contract — embedded-metadata format, SQLite schema, slugify rules, determinism semantics.