CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

qa-analysis

Analyze a GitHub pull request for risk level and generate concrete QA recommendations. Accepts a PR URL or "owner/repo#number" reference. Uses `gh` CLI to fetch the diff and metadata, computes blast radius, scores six risk dimensions, and returns a structured JSON risk assessment. Use when the user invokes /qa-analysis:qa-analysis with a GitHub PR URL or reference, or asks for a PR risk assessment, QA recommendations, or "what should I test?" for a given pull request.

Quality

90%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

SKILL.md
Quality
Evals
Security

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A strong, lean instruction-only skill: executable gh commands, exact output schema, tiered scoring criteria, and a well-sequenced analysis process with override and calibration checkpoints. The main weaknesses are the unspecified composite-score weights (hurting reproducibility of the headline number), a redundant Role & Purpose section, and a 185-line body whose detailed scoring tiers could be split into a reference file.

Suggestions

Specify the explicit dimension weights for the composite risk_score (e.g., data_integrity 0.30, security_surface 0.25, others summing the rest) instead of only stating 'Data integrity and security carry more weight', so the headline numeric output is reproducible across runs.

Trim the 'Role & Purpose' section, which nearly restates the frontmatter description, keeping at most the one-line calibration framing.

Either move the Data Integrity scoring tiers (and optionally the per-area path classification) into a references/ file linked from the body, or add a final validation step before returning (all six dimensions present, overrides and calibration safeguard applied, output is valid JSON).

DimensionReasoningScore

Conciseness

The body is dense and assumes Claude's competence — no explanation of what a PR or `gh` is, no library comparisons — with every scoring rule earning its place. It falls just short of anchor 5 because of minor trimmable material: the 'Role & Purpose' paragraph nearly verbatim restates the frontmatter description, and 'Think like a senior QA engineer: what would keep you up at night?' is flavor. Not anchor 3, since there is no genuinely unnecessary explanation, only these minor instances.

4 / 5

Actionability

Guidance is mostly executable: copy-paste-ready `gh pr view --json …` and `gh pr diff` commands, exact categorical thresholds, tiered data-integrity criteria, and a complete output JSON schema. It misses anchor 5 on one real gap: the core numeric output is 'a weighted overall score' where 'Data integrity and security carry more weight' — the actual dimension weights are never given, so the executor must invent them and results are not reproducible. Not anchor 3, since nothing is pseudocode or high-level hand-waving; the gap is specific and minor relative to the whole.

4 / 5

Workflow Clarity

A clear, correctly ordered 7-step process with an explicit pre-compute stage ('Blast Radius (pre-compute before scoring)') and substantive checkpoints: hard HIGH overrides, a calibration safeguard ('regression_surface ≥ 5 AND complexity ≥ 4 AND data_integrity ≤ 2 → MEDIUM'), and a truncated-diff fallback rule. It misses anchor 5 because there is no final validation of the emitted result (e.g., verifying all six dimensions are present, overrides were applied, and the JSON is well-formed before returning). Not anchor 3, since checkpoints are explicit, not merely implied.

4 / 5

Progressive Disclosure

Scored against the actual bundle (SKILL.md only, no references/scripts/assets): sections are well-organized, self-contained, and nothing is buried or nested. It does not reach anchor 5 because that anchor (and the under-50-line simple-skill exception) doesn't apply — at ~185 lines, the ~30-line Data Integrity scoring tiers and the detailed analysis heuristics are candidates for a one-level-deep reference file. Not anchor 3, since the structure is good and all present content is appropriately placed; only a minor split opportunity exists.

4 / 5

Total

16

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A model description: third-person voice, concrete capability list covering fetch/compute/score/output, input formats (URL and owner/repo#number) named, and an explicit 'Use when…' clause with multiple natural trigger phrases including a colloquial one. Nothing vague or over-claimed.

DimensionReasoningScore

Specificity

The description lists multiple concrete actions — 'fetch the diff and metadata' via `gh` CLI, 'computes blast radius', 'scores six risk dimensions', 'returns a structured JSON risk assessment', 'generate concrete QA recommendations' — with comprehensive coverage of the skill's behavior, matching the anchor-5 pattern rather than the minor-gaps pattern of anchor 4.

5 / 5

Completeness

It explicitly answers both what (analyze a PR for risk level, compute blast radius, score six dimensions, return structured JSON with QA recommendations) and when ('Use when the user invokes /qa-analysis:qa-analysis with a GitHub PR URL or reference, or asks for a PR risk assessment, QA recommendations, or "what should I test?"'), mirroring the anchor-5 example structure exactly. Not anchor 4, since the 'when' clause is fully explicit with concrete trigger phrases.

5 / 5

Trigger Term Quality

Natural user phrasings are comprehensively covered with synonyms: 'GitHub pull request', 'PR URL or "owner/repo#number" reference', 'PR risk assessment', 'QA recommendations', and the colloquial '"what should I test?"'. No commonly-used variation for this task is missing, so it fits anchor 5 rather than the 'a few natural terms missing' of anchor 4.

5 / 5

Distinctiveness Conflict Risk

It occupies a clear niche (PR risk/QA assessment) with distinct triggers — 'risk assessment', 'QA recommendations', 'what should I test' — that are unlikely to capture requests meant for a general code-review or PR-reading skill. Conflict risk is minimal, fitting anchor 5; anchor 4's 'minor overlap risk with closely related skills' would apply if it claimed generic PR review, which it does not.

5 / 5

Total

20

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

mattermost/mattermost-ai-marketplace

Path

plugins/qa-analysis/skills/qa-analysis/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.