ultraqa

Adversarial dynamic e2e QA workflow - generate hostile scenarios, test, verify, fix, report, and clean up

Quality

68%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Low

Low-risk findings worth noting

Fix and improve this skill with Tessl

tessl review fix ./plugins/oh-my-codex/skills/ultraqa/SKILL.md

Quality

Content

77%

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The content is highly actionable with a clear, validated workflow and concrete commands, but it is a long single-file document with some repetition and inline material that could be split into referenced bundle files. Tightening redundancy and offloading the report template / scenario catalog would improve token efficiency and progressive disclosure.

Suggestions

Move the static Structured Report template and the full Required Scenario Matrix class catalog into reference files (e.g. REPORT_TEMPLATE.md, SCENARIO_MATRIX.md) and link them from the body to reduce inline length.

De-duplicate the hostile/edge-case class list, which appears in both 'Required Scenario Matrix' and 'Important Rules', keeping one canonical source.

Trim the restated cycle-step descriptions in the Important Rules section since the Cycle Workflow already specifies them.

Dimension	Reasoning	Score
Conciseness	The body is content-dense and avoids explaining concepts Claude already knows, but it is lengthy with some repetition (scenario classes restated across the matrix, cycle steps, and Important Rules), so it could be tightened rather than a lean level 3.	2 / 3
Actionability	Provides fully executable guidance — exact `omx state ...` commands, literal exit strings, copy-paste prompt blocks for qa-tester/architect/executor, and concrete harness guardrails (env -u OMX_ROOT, pathToFileURL).	3 / 3
Workflow Clarity	The 8-step cycle is explicitly sequenced with validation checkpoints (CHECK RESULT YES/NO criteria), a diagnose→fix→retest feedback loop, bounded retries (max 5 cycles, 3x same failure), and an exit-conditions table — matching the clear-sequence-with-validation anchor.	3 / 3
Progressive Disclosure	The file is well-sectioned with no external references (none present in references/scripts/assets), but it is a single ~260-line monolith where the scenario-matrix class catalog, report template, and state-tracking snippets could be offloaded; not a 3 because content that could be split remains inline.	2 / 3
	Total	10 / 12 Passed

Description

60%

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific and action-oriented but omits any explicit "when to use" trigger guidance and leads with technical jargon rather than natural user language. Adding a Use-when clause with everyday QA/test keywords would lift completeness and trigger quality.

Suggestions

Append a 'Use when ...' clause naming natural triggers, e.g. 'Use when the user asks to stress-test behavior, find edge-case bugs, or run adversarial end-to-end QA beyond a normal test/lint/build pass.'

Soften the jargon lead by including common variations users would actually say (QA, end-to-end tests, test coverage, edge cases) alongside 'adversarial dynamic e2e'.

Tighten distinctiveness by scoping triggers to behaviors a plain test/build skill would not cover, e.g. 'when you need hostile inputs, prompt-injection probes, or misleading-success-output checks.'

Dimension	Reasoning	Score
Specificity	Lists multiple concrete actions — "generate hostile scenarios, test, verify, fix, report, and clean up" — matching the anchor for multiple specific concrete actions.	3 / 3
Completeness	Clearly states what the skill does (the action chain) but provides no "Use when..." trigger clause, so per the guidelines completeness is capped at 2.	2 / 3
Trigger Term Quality	Includes some relevant terms (test, verify, fix, report) but leads with jargon ("adversarial dynamic e2e QA") and omits common natural variations like "QA", "end-to-end tests", or "test coverage"; not a level 3 because coverage of natural user phrasing is incomplete.	2 / 3
Distinctiveness Conflict Risk	The adversarial-e2e-QA niche is fairly distinct, but without explicit trigger scoping it could overlap with a generic testing/verification skill; not a 3 because the triggers do not carve out a clearly non-conflicting niche.	2 / 3
	Total	9 / 12 Passed

Validation

100%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository: Yeachan-Heo/oh-my-codex
Path: plugins/oh-my-codex/skills/ultraqa/SKILL.md
Commit: 435d4a9

Reviewed: 2 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.