experimental-design
Best practices for designing reproducible ML experiments. Use when planning ablations, baselines, or controlled experiments.
72
Quality
88%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
View guide
Securityby
Passed
No findings from the security scan
Experimental Design Best Practice
- ALWAYS include meaningful baselines (not just random):
- At least one classical method baseline
- At least one recent SOTA method baseline
- A simple-but-strong baseline (e.g., linear probe, k-NN)
- Use MULTIPLE random seeds (minimum 3, ideally 5)
- Report mean +/- std across seeds
- Design ablations that isolate EACH key component:
- Remove one component at a time
- Each ablation must be meaningfully different from baseline
- Control variables: change only ONE thing per comparison
- Use standard splits (train/val/test) — never test on training data
- Report wall-clock time and memory usage alongside accuracy
- Repository
- aiming-lab/AutoResearchClaw
- Commit
e2e23c9
- Last updated
- First committed
Is this your skill?
If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.