Designing Experiments

Helps choose and specify a research design before data analysis starts. This skill owns study-design decisions: what is treated, what is compared, what outcome is measured, which assumptions are required, which validation or recovery experiment should follow a failed scientific experiment, and which design is defensible.

It does not fit causal models, estimate treatment effects, interpret fitted model output from existing data, or debug software/build failures.

Decision Framework

1. Control Group?

   • Yes: Go to Step 2.
   • No: Consider Interrupted Time Series (ITS).

2. Unit Structure?

   • Single Treated Unit:
     • With multiple controls: Synthetic Control (SC).
     • No controls: ITS.
   • Multiple Treated Units:
     • With control group: Difference-in-Differences (DiD).

3. Time Structure?

   • Panel Data (Multiple units over time): Required for DiD and SC.
   • Time Series (Single unit over time): Required for ITS.

Method Quick Reference

• Difference-in-Differences (DiD): Compares trend changes between treated and control groups. Assumes Parallel Trends.
• Interrupted Time Series (ITS): Analyzes trend/level change for a single unit after intervention. Assumes Trend Continuity.
• Synthetic Control (SC): Constructs a synthetic counterfactual from weighted control units. Assumes Convex Hull (treated unit within range of controls).

Failed Experiment Recovery

When a scientific experiment or optimization plan produces weak or contradictory results, use the same design surface to:

• Separate implementation or measurement errors from design-assumption failures.
• Identify which assumption should be tested next.
• Define a minimal validation experiment before abandoning the approach.
• State the decision rule for continuing, revising, or stopping the line of work.