CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

data-warehouse-experimentation

Running experiments out of the data warehouse instead of via dedicated experiment platforms. SQL-based assignment, exposure logging discipline, metric definitions in dbt models, statistical analysis in SQL or Python, variance reduction with CUPED, sequential testing, and the operational tradeoffs vs platforms like Statsig and Optimizely. Triggers on warehouse-native experimentation, run experiments in BigQuery, run experiments in Snowflake, dbt experiments, SQL t-test, CUPED variance reduction, exposure log, sample ratio mismatch, sequential testing, mSPRT, doubly robust estimation, build vs buy experimentation. Also triggers when the team is choosing between platform and warehouse, building warehouse-native experiment infrastructure, auditing one, or running an experiment with a custom metric the platform cannot handle.

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

The canonical home for this skill is data-warehouse-experimentation in rampstackco/claude-skills

SKILL.md
Quality
Evals
Security

Quality

Content

71%Weight 40%Scale 1-5

Reviews the quality of instructions and guidance provided to agents. Good implementation is clear, handles edge cases, and produces reliable results.

A well-structured body with excellent progressive disclosure and mostly executable templates, weakened by notable redundancy — the exposure traps, decision factors, and section guidance each appear two to three times across the intro, pitfalls, 12-consideration framework, and closing. Tightening the repetition would move it close to top marks without losing substance.

Suggestions

Consolidate the repeated material: the delayed-exposure/always-fire traps are covered in both 'The exposure log' and 'Common pitfalls', and the platform-vs-warehouse factors appear in the intro, the decision section, and the closing — keep each point in one place and cross-reference the section or reference file instead of restating it.

Cut the duplication between the intro's 'When to use this skill' sentence, the 'What this skill is for' section, and the 12-consideration framework, which re-describe the same scope and guidance three times; state the scope once and let the framework carry the checklist.

Replace the placeholder calls in the Python snippets ('warehouse.query(...)' in the analysis example, 'pre_period_revenue(user_id)' in the CUPED example) with concrete, runnable equivalents (e.g., a BigQuery/Snowflake client call or pandas read_sql) so the templates are copy-paste ready.

DimensionReasoningScore

Conciseness

Mostly efficient and rich in non-obvious discipline (exposure firing rules, salt conventions, realistic-MDE calibration), but the same content recurs across sections: the delayed-exposure and always-fire traps appear in both 'The exposure log' and 'Common pitfalls', the platform-vs-warehouse factors appear in three places (intro, decision section, closing), and the 12-consideration framework restates earlier sections' guidance. This 'could be tightened' redundancy fits anchor 3 better than anchor 4's 'minor instances', though it is well above anchor 2 since none of it is generic explanation of concepts Claude already knows.

3 / 5

Actionability

The SQL is copy-paste ready (FARM_FINGERPRINT assignment, full Welch's t-test, dbt model with ref/var) and the statsmodels power snippet is executable, matching 'mostly executable with minor gaps'. Not anchor 5 because two Python snippets hinge on placeholder calls — 'df = warehouse.query(...)' and 'pre_period_revenue(user_id)' — that are not runnable as written; not anchor 3 because those placeholders are clearly labeled template scaffolding rather than pseudocode hiding key steps.

4 / 5

Workflow Clarity

The architecture section sequences the four components in data-flow order and the 12-consideration framework acts as a design checklist, with real validation checkpoints woven in ('Check the SRM before computing any metric', 'first action is to audit the exposure log and metric definitions before celebrating', pre-registered stop criteria). Not anchor 5 because these checkpoints are distributed across sections rather than an explicit validate-and-recover procedure; this is analysis, not a destructive/batch operation, so the cap at 3 does not apply.

4 / 5

Progressive Disclosure

All 8 referenced files exist and are one level deep (verified against the bundle: the only cross-file mention is a lateral comment pointer in metric-definitions-in-dbt.md to statistical-analysis-templates.md, not a chain). Inline content keeps the key patterns while detail is deferred with one-line descriptions per link, plus a summary reference index — matching the 'clear overview with well-signaled one-level-deep references' anchor. Not below 5: nothing that belongs in a reference is inlined at length, and navigation is easy.

5 / 5

Total

16

/

20

Passed

Description

100%Weight 40%Scale 1-5

Based on the skill's description, can an agent find and select it at the right time? Clear, specific descriptions lead to better discovery.

An exemplary description: concrete capabilities, an explicit and natural trigger list including tool names, and a clear what/when pair, all scoped to a well-defined niche. It is long but every clause is load-bearing, so density does not read as padding.

DimensionReasoningScore

Specificity

The description lists multiple specific concrete actions — 'SQL-based assignment, exposure logging discipline, metric definitions in dbt models, statistical analysis in SQL or Python, variance reduction with CUPED, sequential testing' — with named techniques (CUPED, mSPRT) and named platforms (Statsig, Optimizely), matching the comprehensive-coverage anchor. It is dense rather than padded, so the verbosity penalty does not apply.

5 / 5

Completeness

Both halves are explicit: 'what' is the opening capability list and operational framing ('Running experiments out of the data warehouse instead of via dedicated experiment platforms'), and 'when' is given twice with concrete trigger phrases ('Triggers on warehouse-native experimentation… Also triggers when the team is choosing between platform and warehouse, building warehouse-native experiment infrastructure, auditing one…'). This matches the score-5 example structure exactly; score 4 would require the 'when' to be merely present rather than explicit and concrete.

5 / 5

Trigger Term Quality

The trigger list covers natural phrases a user would actually say — 'run experiments in BigQuery', 'run experiments in Snowflake', 'dbt experiments', 'SQL t-test', 'build vs buy experimentation' — plus synonyms and specialist terms (CUPED variance reduction, exposure log, sample ratio mismatch). Not below 5: no missing common variations of consequence; not above since 5 is the anchor maximum.

5 / 5

Distinctiveness Conflict Risk

The niche is clearly defined — warehouse-native experimentation versus vendor platforms — and anchored by unmistakable triggers ('run experiments in BigQuery', 'dbt experiments', 'build vs buy experimentation') that no generic experimentation skill would claim. Some methodology terms (sequential testing, CUPED) are shared with general stats skills, but the opening sentence scopes everything to the warehouse context, so conflict risk is minimal; this fits the 'clear niche with distinct triggers' anchor better than the 'minor overlap risk' anchor at 4.

5 / 5

Total

20

/

20

Passed

Validation

100%

Checks the skill against the spec for correct structure and formatting. All validation checks must pass before discovery and implementation can be scored.

Validation — 16 / 16 Passed

Validation for skill structure

No warnings or errors.

Repository

rampstackco/claude-skills

Path

dist/codex/.agents/skills/data-warehouse-experimentation/SKILL.md

Reviewed

Table of Contents

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.