CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

vally-eval

Author, validate, and run Vally evaluation suites for agent skills. TRIGGERS: create eval, write eval, add eval, run eval, validate eval, vally eval, eval.yaml, add stimulus, map test to eval, migrate test to eval, eval graders, eval scoring, add eval to CI.

Quality

85%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

SKILL.md
Quality
Evals
Security

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A solidly actionable, well-organized instruction body: every workflow phase has an executable command, validation and re-grade feedback loops are present, and detail is properly deferred to real reference files. All four dimensions land at 4 due to minor gaps — an indirect custom-grader example, a malformed wrapped command, a stale path note, and one second-level reference chain.

Suggestions

Actionability: add a minimal inline custom-grader example instead of deferring entirely to external docs, and fix the wrapped `--grader-plugin` command so it is a single unbroken invocation.

Workflow clarity: correct the stale note that says the relative path "ends at copilot-sdk-executor.ts" when the section is about vally-graders.ts, and state the lifecycle order explicitly (write → validate → run → re-grade) with a gate like "only add suites to CI after validation passes".

Progressive disclosure: resolve the second-level reference chain by inlining the needed facts from ../../../../docs/azure-fixture.md into references/azure-fixture.md, and use one consistent link style for repo files.

DimensionReasoningScore

Conciseness

The body is dense and procedural — commands, paths, and file-layout conventions with almost no padding — e.g. "npm run vally validate-stimulus", "npx @microsoft/vally-cli grade --eval-spec ... --verbose < results/<test-run-name>/results.jsonl". Minor trimmable instances exist ("In most cases, you would like to use a command like this", the dashboard infrastructure explanation), so not 5; not 3 because there is no noticeably verbose or redundant section.

4 / 5

Actionability

Concrete copy-paste commands with cwd comments, exact file layouts ("<repo-root>/evals/<plugin-dirname>/<skill-name>/*.yaml"), and full re-grade invocations. Not 5 because the custom-grader section only says "follow the examples in the official vally documentation" with no inline example, the wrapped --grader-plugin command reads as a broken two-line snippet, and the trailing note resolves to "copilot-sdk-executor.ts" instead of vally-graders.ts. Not 3 because the given commands are fully executable, not pseudocode or high-level hints.

4 / 5

Workflow Clarity

Sections sequence the lifecycle (write → validate → run locally → run in CI → extend → re-grade → collect results) and include an explicit validation step ("npm run vally validate-stimulus"), a re-grade feedback loop ("keep tuning the grader config in eval.yaml and re-grade ... until the results meet your expectations"), and the earlyTerminate/`completed` grader warning as a checkpoint. Not 5 because the overall ordering is only implied by section order and there is no explicit "only proceed when validation passes" gate; not 3 because validation checkpoints are explicitly present.

4 / 5

Progressive Disclosure

The ~99-line body is a well-sectioned overview that defers detail via clearly signaled, existing one-level references ([azure-fixtures](./references/azure-fixture.md), [ci-test](./references/ci-test.md)) plus the official external docs for the spec schema. Not 5 because references/azure-fixture.md chains a second-level reference to ../../../../docs/azure-fixture.md and the body mixes root-relative (/tests/...) and ascent-relative (../../../tests/...) link styles; not 3 because references are clearly signaled and content is appropriately split.

4 / 5

Total

16

/

20

Passed

Description

91%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong description: it states concrete capabilities in third person and pairs them with an explicit, comprehensive trigger list including synonyms and file extensions. The only weaknesses are minor — a few generic trigger terms that could overlap with other eval tooling, and body-covered capabilities (custom graders, CI, re-grading) not reflected in the action list.

DimensionReasoningScore

Specificity

"Author, validate, and run Vally evaluation suites for agent skills" names the domain and three concrete actions, but coverage has minor gaps — custom graders, re-grading, and CI integration are covered by the skill body yet absent from the description. Not 3 because more than 1-2 concrete actions are listed; not 5 because the action list is not comprehensive for what the skill does.

4 / 5

Completeness

The description clearly answers "what" (author, validate, and run Vally evaluation suites) and "when" via the explicit TRIGGERS clause, which is equivalent to a "Use when..." clause with concrete trigger phrases. Not 4 because the when-guidance is explicit rather than weakly implied.

5 / 5

Trigger Term Quality

The explicit TRIGGERS list provides comprehensive natural-phrase coverage with synonyms ("create eval, write eval, add eval"), a file-extension form ("eval.yaml"), and workflow phrases ("add stimulus, map test to eval, migrate test to eval, eval graders, eval scoring, add eval to CI"). Matches the anchor for comprehensive coverage including synonyms and file extensions.

5 / 5

Distinctiveness Conflict Risk

"vally eval", "eval.yaml", and "add stimulus" carve a clear niche, but generic terms like "run eval" and "eval scoring" could overlap with other evaluation/testing skills. Mostly distinct with minor overlap risk against closely related test tooling skills; not 5 because the eval namespace is shared by many testing skills.

4 / 5

Total

18

/

20

Passed

Validation

93%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

CriteriaDescriptionResult

relative_links

Relative link issues: 5 suspicious

Warning

Total

15

/

16

Passed

Repository

microsoft/GitHub-Copilot-for-Azure

Path

.github/skills/vally-eval/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.