testland/guardrail-metrics-reference
Pure-reference catalog of guardrail-metric methodology for online controlled experiments. Defines guardrail metrics (metrics that must NOT degrade for an experiment to ship, even if the primary metric improves), the standard guardrail set (latency / errors / engagement / opt-out), the relationship to OEC (Overall Evaluation Criterion) per Kohavi et al., and pre-commitment of the metric set. The quantitative evaluation mechanics (per-metric alert/block thresholds, Bonferroni / Benjamini-Hochberg multiple-comparison correction) live in references/. Use when designing the metric set for a new experiment, auditing existing experiment configs, or reviewing experiment results before ship-decisions.
75
94%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
View guide
Passed
No findings from the security scan
tile.json
{
"name": "testland/guardrail-metrics-reference",
"version": "1.1.12",
"summary": "Pure-reference catalog of guardrail-metric methodology for online controlled experiments. Defines guardrail metrics (metrics that must NOT degrade for an experiment to ship, even if the primary metric improves), the standard guardrail set (latency / errors / engagement / opt-out), the relationship to OEC (Overall Evaluation Criterion) per Kohavi et al., and pre-commitment of the metric set. The quantitative evaluation mechanics (per-metric alert/block thresholds, Bonferroni / Benjamini-Hochberg multiple-comparison correction) live in references/. Use when designing the metric set for a new experiment, auditing existing experiment configs, or reviewing experiment results before ship-decisions.",
"private": false,
"skills": {
"guardrail-metrics-reference": {
"path": "SKILL.md"
}
}
}