CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

experiments

Use when adding, changing, or reviewing a VS Code A/B experiment, either an experiment-controlled setting or a treatment read with the assignment service, and when deciding whether and where to log an `experimentTrigger` event for a triggered ExP scorecard.

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

82%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A high-quality conventions skill: concrete APIs, code, timings, and decision tables with explicit validation guidance, and no filler. The main weakness is the 'Assignment routing' section, a dense prose wall of edge-case rules that hurts both conciseness and the overview structure a single-file skill should keep.

Suggestions

Move the 'Assignment routing' detail (host cycling, hedging timings, CSP/CORS notes) into a separate reference file (e.g. references/assignment-routing.md) and keep a 3-4 line summary in SKILL.md, preserving the overview-plus-one-level-reference structure.

Rewrite the remaining routing prose as short bullets ('Try hosts in order: enterprise → business → individual; 1s hedging during discovery only; 5s total deadline') so each rule is scannable instead of buried in run-on sentences.

Add a brief end-to-end checklist for the most common task (e.g. add setting schema → set `experiments.override.<treatment>` locally → verify the default resolves → add trigger → test both arms), including how to confirm a local override took effect.

DimensionReasoningScore

Conciseness

The body is dense with repo-specific facts and never explains concepts Claude already knows, and the trigger sections are tight. Not 5 because the 'Assignment routing' section packs dozens of rules into long run-on prose ('Remember the winning host for the window and try it first on subsequent refreshes without time-based hedging; it fails over only on an error...') that could be bulleted and trimmed, and a few rationale sentences ('Dilution only costs statistical power, while a biased trigger gives wrong results') could be cut. Not 3 because there is no padding or filler anywhere.

4 / 5

Actionability

Guidance is executable throughout: copy-adaptable calls (`logSettingExperimentTrigger(telemetryService, MY_SETTING_ID);`), a concrete local-override example (`"experiments.override.config.chat.mySetting.enabled": true`), exact schema snippets (`experiment: { mode: 'auto' }`), specific APIs (`IWorkbenchAssignmentService.getTreatment<T>`, `IExperimentalSettingsService.requestKeys`), concrete hosts/timings, and a named test service (`TestExperimentTriggerTelemetryService`). The common cases are covered with real code and a correct-vs-incorrect ordering example.

5 / 5

Workflow Clarity

Decision flows are clear: three control patterns each with 'use this when' guidance, a when-to-use/when-to-fire structure, a decision table, a pitfalls checklist, and explicit validation ('Test each trigger in every arm with `TestExperimentTriggerTelemetryService`'; 'start a new experiment iteration'). Not 5 because there is no end-to-end sequence for the primary task (add experiment → verify override locally → add trigger → test), and the local-override path gives no way to confirm the assignment took effect.

4 / 5

Progressive Disclosure

No bundle files exist; the body is a single well-sectioned SKILL.md whose two external links ([experimentTrigger.ts], [copilot-instructions.md]) are clearly signaled and one level deep. Not 5 because the body exceeds 50 lines and the ~450-word 'Assignment routing' architectural detail is inlined reference material that would serve better as a separate file, leaving SKILL.md an overview.

4 / 5

Total

17

/

20

Passed

Description

83%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, third-person, trigger-first description that names concrete mechanisms and exact identifiers for a well-defined niche. The 'when' guidance is exemplary; the only weaknesses are a few missing domain synonyms and an implied rather than stated capability clause.

DimensionReasoningScore

Specificity

The description enumerates multiple concrete actions — 'adding, changing, or reviewing a VS Code A/B experiment', 'an experiment-controlled setting', 'a treatment read with the assignment service', and logging 'an `experimentTrigger` event for a triggered ExP scorecard' — with named mechanisms covering the skill's full scope. Not 4 because coverage is comprehensive rather than having minor gaps; keyed object settings are a sub-variant of the experiment-controlled setting already named.

5 / 5

Completeness

The 'when' is explicit and highly specific ('Use when adding, changing, or reviewing a VS Code A/B experiment... and when deciding whether and where to log an `experimentTrigger` event'), and the enumerated tasks imply the what. Not 5 because the capability side is implied by the trigger clauses rather than stated as a distinct 'what this skill does' statement; not 3 because explicit trigger guidance is present and the what is clearly derivable.

4 / 5

Trigger Term Quality

Good natural keyword coverage for the domain: 'A/B experiment', 'experiment', 'treatment', 'assignment service', 'experimentTrigger', 'scorecard', 'ExP' — terms a contributor would actually say. Not 5 because common synonyms like 'flight'/'flighting', 'rollout', or 'graduating an experiment' (used in the body) are missing.

4 / 5

Distinctiveness Conflict Risk

A clear niche with distinct triggers — 'VS Code A/B experiment', 'experiment-controlled setting', 'treatment read with the assignment service', 'triggered ExP scorecard' — that would not fire for unrelated skills. Overlap risk with a generic telemetry skill is minimal given the experiment-specific framing.

5 / 5

Total

18

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

relative_links

Relative link issues: 2 suspicious

Warning

Total

15

/

16

Passed

Repository

microsoft/vscode

Path

.github/skills/experiments/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.