Fair Evaluation — Comparing kane-cli to Other Approaches

When you compare kane-cli to anything else (an agent that generates Playwright scripts, a record-and-replay tool, hand-written scripts, a runtime browser agent), the comparison is only honest if it is like-for-like across the test lifecycle. Most bad comparisons come from mixing phases or zeroing out one side's costs.

The core rule: never compare across lifecycle phases

kane-cli has two phases with very different costs:

• Authoring (first run, or after an edit): an AI agent works out each step → spends LLM tokens. One-time per step.
• Replay (every later run, for each unchanged step that has a recording): executes the recorded action plan from local cache → mostly avoids the authoring cost, runs in seconds. A replay that heals re-authors the failing steps, so it costs more. Cost is proportional to how often a test changes, not how often it runs. (In a shared project, each folder that pulls a test authors it once, on its first run there: shared-projects.md §7a.)

The most common evaluation error is pitting kane-cli authoring against the other approach's execution. That is apples-to-oranges. Compare the same phase on both sides:

So a fair total-cost comparison is:

kane: authoring + re-author-on-change + vision-only replay tokens vs. other: script creation + locator-break repair + ongoing maintenance + any per-run judge

Compare kane authoring to the other approach's creation effort (the tokens/time to produce the script) — not to its execution.

Mandatory corrections (real errors seen under time pressure)

• ❌ "kane spends LLM tokens on every test run." → Wrong. The authoring cost is spent when a step is authored (first run, after an edit, or when a replay heals). Replay of unchanged steps mostly avoids it. State this before any cost claim.
• ❌ "The Playwright scripts already exist, so their creation cost is $0." → Sunk-cost trap. The creation effort was still spent (tokens or engineer hours). For an adoption/ROI decision, either count creation on both sides, or compare the forward maintenance trajectory — never zero out one side's creation while charging the other's.
• ❌ "Free CI runs vs. tokens per test." → Phase mismatch (execution vs. authoring). Both execute cheaply; the real differentiators are creation effort and maintenance under UI churn.
• ❌ Judging a single isolated run. → The right unit is the suite over its lifetime (many runs + UI changes). kane's advantage widens with run count and churn.

What dominates at scale

For any suite that lives long enough to be edited or to outlast a UI refactor, the durable cost is maintenance under UI churn, not run-time compute. Anchor the comparison there, and measure over create → run → maintain, not on one metric or one run.

What kane-cli is purpose-built for

kane-cli is a dedicated agent specialized for browser testing and control — not a general-purpose agent bolted onto a browser. Hold the task constant and compare it on the jobs it is built for:

• Development-flow testing — a developer (or an AI coding agent) validating that a change actually works in a real browser, inline in the dev loop.
• QA regression + smoke management — authoring once and replaying large regression/smoke suites deterministically, with self-heal on drift.
• Daily use as a browser tool — ad-hoc, one-shot natural-language browser tasks (navigate, fill, extract, verify) as an everyday utility.

A fair comparison evaluates kane-cli against alternatives on these jobs, across the full lifecycle — not on a single number or a single run.