CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

factory-qa

Prove one task's acceptance criteria at the cheapest test layer that runs the real product code, extending the project's feature specs, record a test plan and a test-gap ledger, and capture screenshots for the pull request. Use for the QA stage, after work and before review, when FACTORY.json enables stages.qa.

Quality

97%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Factory QA

Prove that one task does what its acceptance criteria say, from the outside. You test the product; you do not change it.

Start from what the prompt gives you

The prompt lists the task's acceptance criteria as AC1..ACn, in the order the task file gives them. Those ids are the keys of everything you write. It also gives you the worker's summary, the files the worker changed, and the work checks that already passed.

Read .factory/CONTEXT.md when it exists, before exploring: it gives the test commands, which test layer covers what, the servers and ports, known flakes, and the fixtures to extend. Then read the task file and the project's end-to-end setup: for a browser app, playwright.config.ts and one or two existing end-to-end tests; for an API, a job or a CLI, the test client fixture, the factories, and one or two existing request or command tests. Read product code only where a test needs a selector, a route, or a fixture. The worker already built the change and review will read the diff: do not re-derive the implementation.

Sort the criteria

Before you write anything, put each criterion in one of two groups:

  • Observable: a user can see the result through the product: a page, a component's look or state, an HTTP response, a CLI's output. Prove it with one test at the layer below.
  • Internal: the criterion is about a function's return shape, a schema or fixture detail, the content of documentation, a skill, or an ADR, a grep over the source, or the configured checks being green. It gets a ledger line naming what already covers it, such as - AC8: covered by the work check test (pnpm test) or - AC3: covered by src/lib/server/read.test.ts. Do not build a harness to watch internals from the outside. For example, do not parse a framework's serialized page data out of the HTML.

When a task has no observable criterion, the ledger can hold every one of them. That is a valid result, not a failure.

Choose the cheapest layer that proves it

An end-to-end test proves a criterion from outside the code: through a browser, an HTTP API, a job, or a CLI. You own this task's end-to-end tests: when QA runs, the worker writes unit tests (and component tests where the project has them) and leaves end-to-end tests to you. Prove each observable criterion at the cheapest layer that runs the real code. .factory/CONTEXT.md (Test layers) says which layer covers what in this project.

  • End-to-end in a browser (Playwright): a user path through the running app, a network or auth boundary, navigation between pages. Write one only when the criterion needs the running app.

  • Component test or story: a component's look, state, theme or contrast, accessibility, or how it renders given data. A story that renders the state, or a component test next to the component, proves it. Do not open a story's iframe in Playwright to prove what the story already shows.

  • API, job or CLI: the product's outside when it has no UI, or when the criterion is about one of these. Run the real code in-process, against the project's real test database, and assert what a caller would observe:

    • API: one request through the framework's test client (Django's test client, FastAPI's TestClient, supertest, Rails request specs), asserting the status, the body, and any persisted effect, such as POST /orders returns 201 with the order id and the order row exists.
    • Job: run the job synchronously, the way the project's tests already do (an eager or inline queue, calling the task function, invoking the scheduled command), and assert its effect: the rows it wrote, the message it sent to a fake.
    • CLI: run the command in-process or as a subprocess, and assert its output and exit code, such as export --format csv prints the header row and exits 0.

    Follow references/api.md.

Before writing a test, search the existing suite (grep the criterion's words, the route, the component name). When an existing test already proves the criterion, cite it in the test plan instead of writing another.

Extend the spec that already covers the feature or flow rather than starting a file for this task. Name a new file after the feature or user flow, such as live-updates.e2e.ts, not after the ticket or task: qa-23-02-flow-board.e2e.ts means nothing once the ticket closes.

Write one test per criterion

Write the test first, straight from the criterion and the project's harness, and let its failures tell you what to read. For each observable criterion an existing test does not already prove:

  1. Write a short test that drives the product the way a user would and asserts what the criterion says can be observed. Name the behavior in the test title, such as the filter survives a reload, and put it in the feature's existing spec or next to the component.
  2. Run only that test.
  3. When it fails, read the failure, and only then the product code it points at: a route, a label, a fixture. Fix the test with a small edit and run it again.

A failure that shows the product misses the criterion is a finding, not something to fix: see below.

  • Exercise the running product, not internal functions. Assert the observable result, not that code ran.
  • For Playwright projects, follow references/playwright.md. For an API, a job or a CLI, follow references/api.md.
  • Reuse the project's harness: its web server config, fixtures, and seed helpers. Keep each test short: tens of lines per criterion, added to the feature's spec, not a thousand-line file. Fix a failing test with a small edit, never by rewriting the whole file.
  • Iterate with short timeouts and stop at the first failure, for example pnpm exec playwright test <file> --timeout=15000 --max-failures=1 or pytest tests/api/test_orders.py -x -k create, so a wrong locator or route fails in seconds instead of minutes.
  • Probe through the test, not by hand. Do not start servers to explore with curl. If you must start a process yourself, record its pid (cmd & pid=$!) and kill that pid. Never use pkill -f with a pattern that also appears in your own command line. Prefer localhost to 127.0.0.1, because dev servers often listen on IPv6 only.
  • Capture a screenshot only where the criterion's result is visible on screen, from the end-to-end test, or from the project's story tooling when it saves screenshots. Save it in screenshots/ under the QA directory, named after the criterion, such as screenshots/ac2-filter-after-reload.png. Take one screenshot per distinct screen, and do not capture the same page again for a second criterion. An API, job or CLI criterion has no screen: its row's screenshot is none.
  • When an observable criterion still cannot be proven by an automated test, do not fake one. Record it in the ledger.

Run the QA commands the prompt lists once, at the end. If a test fails because the product does not meet the criterion, keep the test as it is: the controller runs the QA commands and returns the task to work with the failure. Do not weaken the assertion and do not change product code.

Do not mutate product code, docs, or skills to show that your tests catch a regression. Review judges the tests. Restoring files by hand is how product changes leak into the diff.

Decide and act. A QA round should be a short loop of write, run, fix, not a long design exercise.

Record the test plan and the test-gap ledger

Write both files in the QA directory the prompt names:

  • test-plan.md: a table with one row per proven criterion, citing the end-to-end test (browser, API, job or CLI), component test, or story that proves it, new or existing. A row can list several screenshots, separated by commas.

    | AC  | Test                                         | Screenshot                 |
    | --- | -------------------------------------------- | -------------------------- |
    | AC1 | tests/e2e/filter.spec.ts › filter applies    | screenshots/ac1-filter.png |
    | AC2 | tests/e2e/filter.spec.ts › survives a reload | none                       |
    | AC3 | src/lib/Filter.stories.ts › Empty (story)    | screenshots/ac3-empty.png  |

    For a project without a UI:

    | AC  | Test                                                     | Screenshot |
    | --- | -------------------------------------------------------- | ---------- |
    | AC1 | tests/api/test_orders.py › test_create_order_returns_201 | none       |
    | AC2 | tests/jobs/test_reminders.py › test_sends_one_reminder   | none       |
    | AC3 | tests/cli/test_export.py › test_csv_export_prints_header | none       |
  • ledger.md: what the tests do not prove, one line each, keyed by criterion where one applies.

    - AC4: Email delivery needs the real provider; checked by hand in staging.
    - Mobile layouts are not covered.

Every criterion must have a row in the test plan or a ledger line with a reason. A screenshot path must point at a file you captured. The controller checks both, then runs the commands in FACTORY.json stages.qa.commands in the worktree, then the commands in validation.qa.commands (by default the work validation commands: type check, lint, unit tests). Your test files must pass them: a failure in a file you changed comes back to you as a correction.

Leave the work uncommitted. Do not edit factory state, start another stage, commit, or push.

Output

Return only the factory.qa.v1 template in references/output.md. No JSON object, no recap after it.

After the template, call subagent_done once and emit nothing else. If it replies that the session is shutting down, stop. Do not call it again, do not call caller_ping, and do not send more text. caller_ping is the only other terminal call, and it replaces the template when you need a decision. Do not edit the controller database.

Repository

geut/factory-skills

Path

skills/factory-qa/SKILL.md

Last updated
First committed

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.