Content
75%Weight 40%Scale 1-5Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.
A solidly actionable, well-organized instruction body: every workflow phase has an executable command, validation and re-grade feedback loops are present, and detail is properly deferred to real reference files. All four dimensions land at 4 due to minor gaps — an indirect custom-grader example, a malformed wrapped command, a stale path note, and one second-level reference chain.
Suggestions
Actionability: add a minimal inline custom-grader example instead of deferring entirely to external docs, and fix the wrapped `--grader-plugin` command so it is a single unbroken invocation.
Workflow clarity: correct the stale note that says the relative path "ends at copilot-sdk-executor.ts" when the section is about vally-graders.ts, and state the lifecycle order explicitly (write → validate → run → re-grade) with a gate like "only add suites to CI after validation passes".
Progressive disclosure: resolve the second-level reference chain by inlining the needed facts from ../../../../docs/azure-fixture.md into references/azure-fixture.md, and use one consistent link style for repo files.
| Dimension | Reasoning | Score |
|---|---|---|
Conciseness | The body is dense and procedural — commands, paths, and file-layout conventions with almost no padding — e.g. "npm run vally validate-stimulus", "npx @microsoft/vally-cli grade --eval-spec ... --verbose < results/<test-run-name>/results.jsonl". Minor trimmable instances exist ("In most cases, you would like to use a command like this", the dashboard infrastructure explanation), so not 5; not 3 because there is no noticeably verbose or redundant section. | 4 / 5 |
Actionability | Concrete copy-paste commands with cwd comments, exact file layouts ("<repo-root>/evals/<plugin-dirname>/<skill-name>/*.yaml"), and full re-grade invocations. Not 5 because the custom-grader section only says "follow the examples in the official vally documentation" with no inline example, the wrapped --grader-plugin command reads as a broken two-line snippet, and the trailing note resolves to "copilot-sdk-executor.ts" instead of vally-graders.ts. Not 3 because the given commands are fully executable, not pseudocode or high-level hints. | 4 / 5 |
Workflow Clarity | Sections sequence the lifecycle (write → validate → run locally → run in CI → extend → re-grade → collect results) and include an explicit validation step ("npm run vally validate-stimulus"), a re-grade feedback loop ("keep tuning the grader config in eval.yaml and re-grade ... until the results meet your expectations"), and the earlyTerminate/`completed` grader warning as a checkpoint. Not 5 because the overall ordering is only implied by section order and there is no explicit "only proceed when validation passes" gate; not 3 because validation checkpoints are explicitly present. | 4 / 5 |
Progressive Disclosure | The ~99-line body is a well-sectioned overview that defers detail via clearly signaled, existing one-level references ([azure-fixtures](./references/azure-fixture.md), [ci-test](./references/ci-test.md)) plus the official external docs for the spec schema. Not 5 because references/azure-fixture.md chains a second-level reference to ../../../../docs/azure-fixture.md and the body mixes root-relative (/tests/...) and ascent-relative (../../../tests/...) link styles; not 3 because references are clearly signaled and content is appropriately split. | 4 / 5 |
Total | 16 / 20 Passed |