data-warehouse-experimentation

Running experiments out of the data warehouse instead of via dedicated experiment platforms. SQL-based assignment, exposure logging discipline, metric definitions in dbt models, statistical analysis in SQL or Python, variance reduction with CUPED, sequential testing, and the operational tradeoffs vs platforms like Statsig and Optimizely. Triggers on warehouse-native experimentation, run experiments in BigQuery, run experiments in Snowflake, dbt experiments, SQL t-test, CUPED variance reduction, exposure log, sample ratio mismatch, sequential testing, mSPRT, doubly robust estimation, build vs buy experimentation. Also triggers when the team is choosing between platform and warehouse, building warehouse-native experiment infrastructure, auditing one, or running an experiment with a custom metric the platform cannot handle.

Quality

89%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide

Securityby

Passed

No findings from the security scan

Quality

Content

78%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

The body is highly actionable with executable SQL/Python throughout, strong progressive disclosure into eight real reference files, and a clear workflow framework with validation disciplines. Its main weakness is conciseness: narrative framing and restated intuitions add tokens that a competent reader does not need.

Suggestions

Tighten the narrative framing prose (e.g., 'The platforms are good.', 'The honest version.', 'This is the most powerful variance reduction technique...') into imperative statements that assume the reader already knows the domain.

Move some of the repeated intuitions (CUPED 'predict the metric from pre-experiment data' intuition appears in both the body and is restated in the power/sequential sections) to the reference files to avoid re-explaining concepts.

In the 12-consideration framework, pair each checkpoint with the single command or query that executes or verifies it (e.g., the SRM check SQL inline at step 5) so the workflow is a runnable checklist rather than a list of principles.

Dimension	Reasoning	Score
Conciseness	The body is mostly efficient and assumes competence, but includes some unnecessary framing prose ('The platforms are good.', 'This is the most powerful...', 'The honest version.') and contextual narration that could be trimmed without losing actionability, placing it just below the 'efficient' anchor.	3 / 5
Actionability	Provides fully executable, copy-paste-ready SQL and Python blocks for hash assignment, exposure logging, Welch's t-test, CUPED, and power analysis, with the common cases covered and inline notes on edge cases (proportions tests, non-parametric, UDF conversion).	5 / 5
Workflow Clarity	The 12-consideration framework and the four-component architecture give a clear sequence, and destructive/batch-relevant disciplines (SRM check, single-fire exposure, validation of metric definitions) are present as explicit checkpoints; only a few checkpoints are stated as principles rather than verified steps, so it sits just below the top anchor.	4 / 5
Progressive Disclosure	SKILL.md is a lean overview with eight well-signaled, one-level-deep references in a dedicated Reference files section, all of which exist as real files; detailed SQL templates, pitfalls, and math are split into references rather than inlined, giving clear navigation.	5 / 5
	Total	17 / 20 Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

The description is specific, comprehensive, and explicitly pairs 'what' with two 'when' trigger clauses that include natural phrasings and product names. It occupies a distinct niche with low conflict risk against adjacent experimentation skills. It is long but every clause earns its place; minor verbosity is the only weakness.

Dimension	Reasoning	Score
Specificity	Lists many concrete capabilities ('SQL-based assignment, exposure logging discipline, metric definitions in dbt models, statistical analysis in SQL or Python, variance reduction with CUPED, sequential testing'), giving comprehensive coverage of the skill's actions rather than vague language.	5 / 5
Completeness	Explicitly answers 'what' (the enumerated capabilities and operational tradeoffs) and 'when' via two explicit trigger clauses ('Triggers on...', 'Also triggers when...'), with concrete trigger phrases.	5 / 5
Trigger Term Quality	Comprehensive natural term coverage including synonyms, product names (BigQuery, Snowflake, Statsig, Optimizely), and specific techniques users would actually say ('run experiments in BigQuery', 'dbt experiments', 'CUPED variance reduction', 'sample ratio mismatch', 'mSPRT').	5 / 5
Distinctiveness Conflict Risk	A clear, specialized niche (warehouse-native experimentation vs vendor platforms) with distinct triggers (dbt experiments, SQL t-test, CUPED, mSPRT) that are unlikely to collide with general experimentation or feature-flag skills.	5 / 5
	Total	20 / 20 Passed

Validation

93%

Warnings & errors only

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 15 / 16 Passed

Validation for skill structure

Criteria	Description	Result
frontmatter_unknown_keys	Unknown frontmatter key(s) found; consider removing or moving to metadata	Warning

	Total	15 / 16 Passed

Repository: rampstackco/claude-skills
Path: skills/data-warehouse-experimentation/SKILL.md
Commit: 0479242

Reviewed: 6 days ago

Table of Contents

Discovery Implementation Validation

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.