Lines of questioning — Beta user research

Point 1 — Understand user needs

Ask:

• Who has used the service in research since private beta opened, and how were they recruited?
• What did you learn from watching them that the feedback form did not show?
• Which cohort is still missing, and what decision are you taking without them?
• What did you change in the service because of a session, and what did the next session show?

Evidence to request: session logs, recruitment criteria, a change linked to a named session.

Answers that indicate Met: repeated first-hand sessions during private beta, not only a survey at the end.

Answers that indicate Not Met: the research pack is feedback forms and analytics only.

Point 2 — Solve a whole problem for users

Ask:

• What task were participants given, and did it include the step your service does not own?
• Where did they get stuck outside the interface (documents, a phone call, a wait for a decision)?
• What did you stop asking users to do, because the whole task was unreasonable?

Evidence to request: task scripts for end-to-end sessions, notes on the handoff.

Answers that indicate Met: participants attempted the real outcome, not a tour of the screens.

Answers that indicate Not Met: testing was a click-through of the happy path.

Point 5 — Make sure everyone can use it

Ask, and do not accept a general answer:

• Which participants used a screen reader, and was it JAWS, NVDA, VoiceOver, or something else?
• Which participants used a screen magnifier, or relied on magnification at a level that changes the layout?
• Which participants used speech input, such as Dragon?
• What failed in those sessions, and what is fixed on the build the panel can use today?
• If nobody using assistive technology was recruited, what was the reason, and why was private beta opened without them?

Evidence to request: session records that name the assistive technology, the participant's setup, the task, and the outcome. An audit report is the technical assessor's evidence. It does not replace these sessions.

Answers that indicate Met: at least one screen reader user (JAWS or NVDA, and you should ask about both communities where the service is aimed at the general public), plus magnification or speech input as the journey requires, on the working Beta, with fixes the team can demonstrate.

Answers that indicate Not Met — red line B1, verdict NOT_MET:

• no participant used assistive technology
• the team tested with 24 general citizens and a feedback form
• the team offers an automated scan, a keyboard check by a developer, or a future booking

When this is the answer, write the finding so it plainly says there was no assistive technology testing, and name JAWS, NVDA, and screen reader as the uncovered screen-reader gap. Mention magnifiers and speech input if they are missing too.

Do not trade the points off

A strong point 1 and point 2 do not lift point 5. Record the red line and move on.