Prove one task's acceptance criteria at the cheapest test layer that runs the real product code, extending the project's feature specs, record a test plan and a test-gap ledger, and capture screenshots for the pull request. Use for the QA stage, after work and before review, when FACTORY.json enables stages.qa.
97%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Passed
No findings from the security scan
Prove that one task does what its acceptance criteria say, from the outside. You test the product; you do not change it.
The prompt lists the task's acceptance criteria as AC1..ACn, in the order the task file gives them. Those ids are the keys of everything you write. It also gives you the worker's summary, the files the worker changed, and the work checks that already passed.
Read .factory/CONTEXT.md when it exists, before exploring: it gives the test commands, which test layer covers what, the servers and ports, known flakes, and the fixtures to extend. Then read the task file and the project's end-to-end setup: for a browser app, playwright.config.ts and one or two existing end-to-end tests; for an API, a job or a CLI, the test client fixture, the factories, and one or two existing request or command tests. Read product code only where a test needs a selector, a route, or a fixture. The worker already built the change and review will read the diff: do not re-derive the implementation.
Before you write anything, put each criterion in one of two groups:
- AC8: covered by the work check test (pnpm test) or - AC3: covered by src/lib/server/read.test.ts. Do not build a harness to watch internals from the outside. For example, do not parse a framework's serialized page data out of the HTML.When a task has no observable criterion, the ledger can hold every one of them. That is a valid result, not a failure.
An end-to-end test proves a criterion from outside the code: through a browser, an HTTP API, a job, or a CLI. You own this task's end-to-end tests: when QA runs, the worker writes unit tests (and component tests where the project has them) and leaves end-to-end tests to you. Prove each observable criterion at the cheapest layer that runs the real code. .factory/CONTEXT.md (Test layers) says which layer covers what in this project.
End-to-end in a browser (Playwright): a user path through the running app, a network or auth boundary, navigation between pages. Write one only when the criterion needs the running app.
Component test or story: a component's look, state, theme or contrast, accessibility, or how it renders given data. A story that renders the state, or a component test next to the component, proves it. Do not open a story's iframe in Playwright to prove what the story already shows.
API, job or CLI: the product's outside when it has no UI, or when the criterion is about one of these. Run the real code in-process, against the project's real test database, and assert what a caller would observe:
TestClient, supertest, Rails request specs), asserting the status, the body, and any persisted effect, such as POST /orders returns 201 with the order id and the order row exists.export --format csv prints the header row and exits 0.Follow references/api.md.
Before writing a test, search the existing suite (grep the criterion's words, the route, the component name). When an existing test already proves the criterion, cite it in the test plan instead of writing another.
Extend the spec that already covers the feature or flow rather than starting a file for this task. Name a new file after the feature or user flow, such as live-updates.e2e.ts, not after the ticket or task: qa-23-02-flow-board.e2e.ts means nothing once the ticket closes.
Write the test first, straight from the criterion and the project's harness, and let its failures tell you what to read. For each observable criterion an existing test does not already prove:
the filter survives a reload, and put it in the feature's existing spec or next to the component.A failure that shows the product misses the criterion is a finding, not something to fix: see below.
pnpm exec playwright test <file> --timeout=15000 --max-failures=1 or pytest tests/api/test_orders.py -x -k create, so a wrong locator or route fails in seconds instead of minutes.curl. If you must start a process yourself, record its pid (cmd & pid=$!) and kill that pid. Never use pkill -f with a pattern that also appears in your own command line. Prefer localhost to 127.0.0.1, because dev servers often listen on IPv6 only.screenshots/ under the QA directory, named after the criterion, such as screenshots/ac2-filter-after-reload.png. Take one screenshot per distinct screen, and do not capture the same page again for a second criterion. An API, job or CLI criterion has no screen: its row's screenshot is none.Run the QA commands the prompt lists once, at the end. If a test fails because the product does not meet the criterion, keep the test as it is: the controller runs the QA commands and returns the task to work with the failure. Do not weaken the assertion and do not change product code.
Do not mutate product code, docs, or skills to show that your tests catch a regression. Review judges the tests. Restoring files by hand is how product changes leak into the diff.
Decide and act. A QA round should be a short loop of write, run, fix, not a long design exercise.
Write both files in the QA directory the prompt names:
test-plan.md: a table with one row per proven criterion, citing the end-to-end test (browser, API, job or CLI), component test, or story that proves it, new or existing. A row can list several screenshots, separated by commas.
| AC | Test | Screenshot |
| --- | -------------------------------------------- | -------------------------- |
| AC1 | tests/e2e/filter.spec.ts › filter applies | screenshots/ac1-filter.png |
| AC2 | tests/e2e/filter.spec.ts › survives a reload | none |
| AC3 | src/lib/Filter.stories.ts › Empty (story) | screenshots/ac3-empty.png |For a project without a UI:
| AC | Test | Screenshot |
| --- | -------------------------------------------------------- | ---------- |
| AC1 | tests/api/test_orders.py › test_create_order_returns_201 | none |
| AC2 | tests/jobs/test_reminders.py › test_sends_one_reminder | none |
| AC3 | tests/cli/test_export.py › test_csv_export_prints_header | none |ledger.md: what the tests do not prove, one line each, keyed by criterion where one applies.
- AC4: Email delivery needs the real provider; checked by hand in staging.
- Mobile layouts are not covered.Every criterion must have a row in the test plan or a ledger line with a reason. A screenshot path must point at a file you captured. The controller checks both, then runs the commands in FACTORY.json stages.qa.commands in the worktree, then the commands in validation.qa.commands (by default the work validation commands: type check, lint, unit tests). Your test files must pass them: a failure in a file you changed comes back to you as a correction.
Leave the work uncommitted. Do not edit factory state, start another stage, commit, or push.
Return only the factory.qa.v1 template in references/output.md. No JSON object, no recap after it.
After the template, call subagent_done once and emit nothing else. If it replies that the session is shutting down, stop. Do not call it again, do not call caller_ping, and do not send more text. caller_ping is the only other terminal call, and it replaces the template when you need a decision. Do not edit the controller database.
geut/factory-skills
97be30f
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.