CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

flaky-smoke-tests

Diagnose intermittent VS Code Electron smoke-test failures from the Azure DevOps Flaky Smoke Tests pipeline (definition 700). Covers finding failed iterations, downloading task logs and platform artifacts with Azure CLI, correlating cumulative runner logs, tracing the introducing commit, and queueing focused validation runs.

70

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Low

Low-risk findings worth noting

The canonical home for this skill is flaky-smoke-tests in microsoft/vscode

SKILL.md
Quality
Evals
Security

Quality

Content

88%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is a strongly executed operational runbook: a clearly sequenced nine-step workflow of executable Azure CLI, git, and npm commands with explicit validation checkpoints and feedback loops, written almost exclusively from non-obvious pipeline-specific knowledge. Its only structural weakness is that everything lives inline in SKILL.md, where some dense diagnostic-interpretation detail could be split into a reference file.

DimensionReasoningScore

Conciseness

Nearly every section conveys pipeline-specific knowledge Claude cannot know (iteration task naming, continueOnError semantics, shared-logs overwrites, attempt-suffixed artifact names) with no filler or explanation of known concepts, but a few passages are over-dense and could be trimmed — e.g., the very long copilot-runtime-logs bullet and the extended pitfall lists in section 6 — matching the 4 anchor ('minor instances of over-explanation that could be trimmed') rather than 5's 'every token earns its place'.

4 / 5

Actionability

Guidance is fully executable and copy-paste ready: exact az devops invoke commands with concrete --query strings, jq/PowerShell log conversion, git show/log/blame commands, a real az pipelines run invocation with platform parameters, and local validation commands, matching the 5 anchor where specific examples cover the common cases.

5 / 5

Workflow Clarity

Steps 1-9 are clearly sequenced (confirm build, find failed iterations, download task log, download artifacts, correlate, interpret, find commit, queue validation, validate locally) with explicit validation checkpoints and feedback loops: confirming the queued run's sourceVersion/templateParameters, re-checking the gesture/dispatch/request/rendering chain on every validation run, and explicit fallback when local dependencies are missing, matching the 5 anchor.

5 / 5

Progressive Disclosure

Sections are well-organized and easy to navigate with the workflow steps carrying the core guidance, but the skill is a single ~280-line file with no bundle references, and dense reference-worthy material (copilot-runtime log capture semantics, UI-diagnostics pitfalls) is inlined, matching the 4 anchor ('most content is appropriately placed; minor organization gaps') rather than 5's well-signaled one-level-deep references.

4 / 5

Total

18

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is highly specific and distinctive, clearly scoping the skill to a single Azure DevOps pipeline and enumerating five concrete diagnostic actions. Its only weakness is that the 'when to use' trigger is implicit in the first sentence rather than an explicit 'Use when...' clause with broader natural synonyms.

DimensionReasoningScore

Specificity

Names the exact domain ("intermittent VS Code Electron smoke-test failures from the Azure DevOps Flaky Smoke Tests pipeline (definition 700)") and lists five concrete actions — "finding failed iterations, downloading task logs and platform artifacts with Azure CLI, correlating cumulative runner logs, tracing the introducing commit, and queueing focused validation runs" — comprehensively covering the diagnostic workflow, matching the 5 anchor rather than 4's 'minor gaps in coverage'.

5 / 5

Completeness

The 'what' is explicit and multi-part, and the 'when' is stated ("failures from the Azure DevOps Flaky Smoke Tests pipeline (definition 700)"), but the trigger is embedded in the capability sentence rather than an explicit "Use when..." clause, matching anchor 4 ('when' could be more explicit) rather than 5's concrete trigger phrases.

4 / 5

Trigger Term Quality

Natural terms include "intermittent", "smoke-test failures", "flaky" (in the pipeline name), and "Azure DevOps", but standalone synonyms users would say such as "flaky tests", "CI failures", or "build failures" are absent, matching the 4 anchor ('good keyword coverage; a few natural terms missing') rather than 5's comprehensive synonym coverage.

4 / 5

Distinctiveness Conflict Risk

The description is pinned to one named pipeline (definition 700) for VS Code Electron smoke tests, giving it a clear niche with distinct triggers and minimal conflict risk with any other skill, matching the 5 anchor.

5 / 5

Total

18

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

posit-dev/positron

Path

.github/skills/flaky-smoke-tests/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.