framework-choice-advisor

Overview

A team is starting a new test automation suite and needs to pick the stack: framework, runner, assertion library, reporter, CI integration, fixture system, parallelisation strategy, retry policy. Auto-scaffolding the whole framework in one shot is the integration-friction failure mode that dominates AI-in-testing adoption (2025 World Quality Report: 37% of teams). The honest deliverable is decision support, not auto-scaffolding.

This skill is a pure reference: a decision tree + tradeoff matrix the team uses as a checklist. It does not generate framework boilerplate. After the team picks a stack, the per-framework skills (playwright-testing, cypress-testing, etc.) document the configuration; this skill stops at "you picked Playwright + Jest, here's the canonical directory layout to use".

When to use

• Greenfield: starting a new test automation suite from zero.
• Migration: the team is moving off a legacy framework (most commonly Selenium to Playwright, whose auto-waiting removes the manual-wait flakiness that dominates Selenium suites, per the Playwright actionability docs).
• Multi-stack consolidation: the team has three frameworks across product areas and is deciding which to standardise on.
• Hiring-driven re-evaluation: the team's skills mix shifted (e.g., from Java to TypeScript) and the framework choice should follow.

Do not use this skill when:

• The framework is already chosen and the team needs configuration / API help - use the per-framework skill (playwright-testing, cypress-testing, etc.).
• The team's question is "should we add E2E vs unit vs contract tests" - that's test-pyramid-balancer.
• Mobile-native vs hybrid app selection - use the qa-mobile plugin's documentation directly.

Step 1 - Frame the decision against the project's NFRs

Six NFR axes drive framework choice. Score each 1 - 5 for the project; rank them by priority. The framework matrix in Step 2 uses these scores.

Step 2 - Framework tradeoff matrix (web E2E)

The 2026-recommendation tree for greenfield web E2E:

• Multi-browser required + multi-language team → Playwright.
• Single-browser (Chromium) + JS-only team → Playwright or Cypress; Cypress has stronger component-testing DX.
• Legacy / migration off existing Selenium suite → either incremental migration to Playwright (preferred) or modernise the Selenium suite via WebdriverIO's runner ergonomics.

Step 3 - Framework tradeoff matrix (other test layers)

Mobile native

Decision: if the team is single-platform native (iOS only or Android only), use the first-party framework. Cross-platform → Appium, accept the WebDriver flake-tax. React-Native specifically → Detox.

API / contract

Performance

Step 4 - Reference directory layouts

After the team has chosen a stack, adopt the canonical directory layout the per-framework skill assumes - a convention, not a mandate, but the starting point a newcomer can read. The layouts for Playwright + Jest, Cypress + Mocha, and Selenium / WebdriverIO are in references/directory-layouts.md.

Step 5 - CI integration patterns

Universal across frameworks:

Step 6 - When to defer the decision

The skill recommends deferring framework choice when:

• The product surface is too new - no stable URL / API to test against.
• The team's required languages aren't decided yet (hiring in progress).
• The team is debating monolith vs micro-frontend; the framework choice depends on the product architecture.

In these cases, the right output is an explicit deferral note: "no decision today; revisit when (a)/(b)/(c) resolves."

Anti-patterns

Limitations

• Coverage is the web / mobile / API / perf canonical set. Specialised layers (desktop apps via Spectron / Tauri, embedded devices, hardware-in-loop) are out of scope.
• Tradeoff matrix is point-in-time. Frameworks evolve quickly; the 2026 ranking will be stale in 18 months. Re-read this skill before every greenfield decision.
• Tradeoffs are illustrative. A specific product can have constraints that flip the recommendation - e.g., a Salesforce internal app where the legacy stack is JVM-only would override the "Playwright by default" guidance.
• Hire-ability is geography-dependent. "Hire-ability" assumes US / EU markets in the matrix; for other markets, the team's local pool is the authority.
• No automated framework scaffolding. Per the introduction, this is intentional - auto-scaffolded boilerplate is the dominant failure mode the research flags. Use the per-framework skill after the decision.
• No closed-platform comparison. Mabl / Testim / Functionize / TestSigma are no-code platforms; they compete with this entire category, not with one framework. If the team is choosing "code vs no-code", that's a strategic decision beyond this skill's scope.

Hand-off targets

• After the framework is chosen, configure it → per-framework skill (playwright-testing, cypress-testing, selenium-testing, webdriverio-testing).
• Tune the layer balance after the framework is in place → test-pyramid-balancer.
• Convert Selenium suite → Playwright (the most common 2026 migration) → migration is a project, not an agent; this skill provides the decision input, the per-framework skills provide the target configuration.

References

• Playwright official documentation - locator hierarchy, trace viewer, parallel sharding: https://playwright.dev/
• Cypress documentation: https://docs.cypress.io/
• WebdriverIO documentation: https://webdriver.io/
• Martin Fowler - Page Object pattern (canonical definition): https://martinfowler.com/bliki/PageObject.html
• Luo et al., "An Empirical Analysis of Flaky Tests" (FSE 2014) - async-wait is the largest flake category (45%), the class Selenium leaves to manual synchronization: https://mir.cs.illinois.edu/marinov/publications/LuoETAL14FlakyTestsAnalysis.pdf
• Playwright actionability docs - built-in auto-waiting and auto-retrying assertions that remove a major Selenium flake source: https://playwright.dev/docs/actionability
• Capgemini World Quality Report 2025-26 - 37% cite integration friction as the dominant AI-in-testing blocker (justifies why this is decision-support, not auto-scaffolding): https://www.capgemini.com/insights/research-library/world-quality-report-2025-26/
• ISTQB glossary - test automation framework: https://glossary.istqb.org/en_US/term/test-automation-framework
• ISTQB glossary - keyword-driven testing (relevant to Karate / Postman DSL choice): https://glossary.istqb.org/en_US/term/keyword-driven-testing
• ISO/IEC 25010 - quality characteristics (used in Step 1 NFR scoring): https://en.wikipedia.org/wiki/ISO/IEC_25010
• playwright-testing, cypress-testing, selenium-testing, webdriverio-testing - downstream per-framework configuration skills.
• test-pyramid-balancer - layer-mix tuning after the framework is in place.