ablation-planner

Use when main results pass result-to-claim (`claim_supported = yes` or `partial`) and ablation studies are needed for paper submission. A secondary Codex agent designs ablations from a reviewer's perspective; the local executor reviews feasibility and implements.

Quality

81%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

75%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured, actionable workflow with clear sequencing and real validation checkpoints (feasibility review, smoke tests). The main gaps are an implicit rather than explicit run-phase feedback loop and inlined template content that could be split out.

Suggestions

Make the run-phase feedback loop explicit: add an instruction like 'if a smoke test fails, diagnose and fix before re-running; do not launch the full ablation until the smoke test passes'.

Move the large Codex spawn_agent prompt template into a referenced file (e.g., references/reviewer_prompt.md) and keep a concise summary inline, improving both conciseness and progressive disclosure.

Add a concrete example of an EXPERIMENT_LOG.md ablation entry (including the negative-result case) so result tracking is copy-paste ready.

Dimension	Reasoning	Score
Conciseness	The body is lean and assumes Claude's competence — it does not explain what ablations are or basic ML concepts, and the spawn_agent template is purposeful. Not a 5 because the Rules section repeats guidance already implied by the workflow steps and could be trimmed slightly.	4 / 5
Actionability	Provides a concrete fill-in spawn_agent prompt template and a structured markdown output template, plus concrete run steps (smoke test, descriptive naming like 'ablation-no-module-X'). Not a 5 because the run phase lacks copy-paste-ready commands for actually launching configs/tracking in EXPERIMENT_LOG.md.	4 / 5
Workflow Clarity	Steps 1–5 are clearly sequenced with checkpoints (Step 4 feasibility review, Step 5 'smoke test each ablation before the full run', record negative results). Not a 5 because the run-phase validate→fix→retry feedback loop is only implicit — there is no explicit 'if smoke test fails, fix and re-test' instruction.	4 / 5
Progressive Disclosure	Well-organized into clear sections (When to Use, Workflow Steps 1–5, Rules) with no nested or buried references, and the content is appropriately self-contained. Not a 5 because at ~120 lines it exceeds the simple-skill threshold and the Codex prompt template is inlined rather than split into a reference file.	4 / 5
	Total	16 / 20 Passed

Description

87%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

A strong, specific description that clearly states both what the skill does and when to use it, with low conflict risk. Its main weakness is leaning on internal tool/output names as triggers rather than the full range of natural user phrasings.

Suggestions

Add a few natural-language trigger synonyms (e.g., 'sensitivity analysis', 'reviewer-requested ablations', 'component attribution') so the skill surfaces when users phrase the need without referencing the result-to-claim tool.

Consider broadening the trigger beyond the internal 'result-to-claim' output state to also fire on direct user requests like 'I need ablations before submission'.

Dimension	Reasoning	Score
Specificity	Names the domain (ablation studies, paper submission) and several concrete actions ('designs ablations', 'reviews feasibility and implements'), but coverage is not fully comprehensive. Not a 5 because it does not enumerate the full set of capabilities (e.g., parse plan, track results).	4 / 5
Completeness	Explicitly answers both 'what' ('A secondary Codex agent designs ablations... the local executor reviews feasibility and implements') and 'when' with concrete trigger phrases ('Use when main results pass result-to-claim... and ablation studies are needed for paper submission'). Clearly matches the 5 anchor.	5 / 5
Trigger Term Quality	Includes natural terms a user would say ('ablation studies', 'paper submission') but the primary trigger leans on internal tool names ('result-to-claim', 'claim_supported = yes or partial') and misses common synonyms like 'sensitivity analysis' or 'reviewer questions'. Not a 5 due to those missing natural variations; not a 3 because several genuinely natural phrases are present.	4 / 5
Distinctiveness Conflict Risk	Occupies a clear niche (designing ablation studies for ML paper submission) with distinct, specific triggers, making accidental invocation by unrelated skills unlikely. Minimal conflict risk.	5 / 5
	Total	18 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
allowed_tools_field	'allowed-tools' contains unusual tool name(s)	Warning

	Total	15 / 16 Passed

Repository: wanshuiyin/Auto-claude-code-research-in-sleep
Path: skills/skills-codex/ablation-planner/SKILL.md
Commit: f4f20f9

Reviewed: 1 day ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.