Content
85%Weight 40%Scale 1-5Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.
An exceptionally actionable, well-sequenced multi-repo runbook: exact paths, runnable verification commands, and hard-won gotchas (the Jira nested-JSON partition_field risk, migration batching) with validation checkpoints after each phase. Its weaknesses are the absence of any progressive disclosure — all detail lives inline in one long SKILL.md — and noticeable repetition of the DynamicSourceSetup/OAuth-genericity message across three sections.
Suggestions
Move the per-repo detail (the posthog/code checklist, the posthog/posthog checklist, and the Tier-1 source catalog table) into references/ files (e.g. references/code-checklist.md, references/scout-checklist.md, references/source-catalog.md), keeping SKILL.md as a lean overview of the three surfaces, deploy ordering, and gotchas.
Consolidate the DynamicSourceSetup / generic-OAuth guidance, which currently appears in 'The setup form', 'OAuth plumbing — NOT needed per source anymore', and 'Setup-form specifics', into a single authoritative section.
Trim background exposition such as the created_via attribution table and the supported OAuth kind list, or move them to a reference file, since they are lookup material rather than steps in the add-a-source workflow.
| Dimension | Reasoning | Score |
|---|---|---|
Conciseness | Nearly every token is repo-specific knowledge Claude cannot know (file paths, payload keys, migration-collision pitfalls, the Jira JSON-blob gotcha) with no padding or explanation of known concepts. However, the 'route to DynamicSourceSetup / generic OAuth flow, no bespoke form' guidance is repeated across three sections ('The setup form', 'OAuth plumbing — NOT needed per source anymore', 'Setup-form specifics') and could be consolidated; not 3 because apart from that repetition nothing is unnecessary, not 5 because that repetition is more than trivial. | 4 / 5 |
Actionability | Fully executable guidance throughout: exact file paths per surface ('packages/ui/src/features/inbox/hooks/useSignalSourceToggles.ts'), runnable commands ('python manage.py makemigrations signals', 'pnpm typecheck', 'node scripts/build.js'), a concrete payload-key table per source, and copy-adapt pointers ('Copy github_issues.py and adapt'). Copy-paste ready and covering the common credential and OAuth cases; not 4 because even edge cases (ssh-tunnel, file-upload, JSON-blob tables) carry specific handling. | 5 / 5 |
Workflow Clarity | The multi-repo process is explicitly sequenced: a 'Deploy ordering (do not skip)' section gates the backend-before-UI dependency, numbered per-repo checklists enumerate every step, and each phase ends with a Verify section ('makemigrations --check is clean', 'pnpm --filter @posthog/shared... build', biome lint) plus migration-collision and merge-queue guidance. Checklists with explicit validation checkpoints match the 5 anchor; not 4 because validation is present after every phase, not just implied. | 5 / 5 |
Progressive Disclosure | The body is a ~256-line monolith with no bundle files at all — the per-repo checklists (posthog/code, posthog/posthog) and the Tier-1 source catalog table are detail material that belongs in references/ files with a lean overview in SKILL.md, per the 'overview pointing to detailed materials' rationale. Internal structure is strong (clear headers, tables, inline cross-links like 'See The self-driving wizard surface below'), so navigation works; not 4 because content that should be separate is inline with zero external references, not 2 because the structure is well-organized, not minimal. | 3 / 5 |
Total | 17 / 20 Passed |