CtrlK
BlogDocsLog inGet started
Tessl Logo
PluginPublic

testland/model-performance-regression-gate

Computes held-out metrics (accuracy, F1, AUC, RMSE) for a retrained model and compares them against the current production model, failing promotion when any metric regresses beyond a configured tolerance. Adds per-segment checks via Deepchecks WeakSegmentsPerformance so a model that improves globally but regresses on a key slice is still blocked. Use when a retrained model is a candidate for promotion and the CI pipeline must enforce a per-metric pass/fail gate before the artifact is pushed to the model registry.

Quality

88%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Overview
Quality
Evals
Security
Files

gate-config-and-metrics.mdreferences/

Gate config and metric reference

Deep detail for model-performance-regression-gate: the externalised YAML threshold config plus its loader, and the per-metric scikit-learn API notes. The runnable gate itself stays in SKILL.md; this file holds the tunables and the full metric signatures.

YAML threshold config

Externalise tolerances so non-engineers can tune them via a PR rather than editing Python:

# config/gate_thresholds.yaml
metrics:
  accuracy:
    tolerance: 0.01
    higher_is_better: true
  f1_weighted:
    tolerance: 0.02
    higher_is_better: true
  roc_auc:
    tolerance: 0.01
    higher_is_better: true
segment:
  max_ratio_change: 0.15
  min_segment_size_ratio: 0.05

Load it into the same TOLERANCES and HIGHER_IS_BETTER structures the gate uses in Step 4:

import yaml

with open("config/gate_thresholds.yaml") as f:
    cfg = yaml.safe_load(f)

TOLERANCES = {k: v["tolerance"] for k, v in cfg["metrics"].items()}
HIGHER_IS_BETTER = {k for k, v in cfg["metrics"].items() if v["higher_is_better"]}

Per-metric scikit-learn signatures

Per scikit-learn model evaluation docs:

  • accuracy_score(y_true, y_pred) - higher better.
  • f1_score(y_true, y_pred, average='weighted') - higher better.
  • roc_auc_score(y_true, y_score) requires probability estimates; for multiclass use average='weighted', multi_class='ovr'.
  • root_mean_squared_error(y_true, y_pred) (regression) is in target units, lower better, so its tolerance direction is inverted in Step 4.

references

gate-config-and-metrics.md

SKILL.md

tile.json