Use for any question or task about data drift or quality monitoring for a production ML model, including a fired drift alert and metrics such as precision that degrade slowly unnoticed: Evidently reports with tests for data drift, target drift and performance regression as a CI gate or scheduled job, and a playbook to pick rollback, retrain, quarantine or alert re-tuning.
91%
Does it follow best practices?
Run evals on this skill
Adds up to 20 points to the overall score
Passed
No findings from the security scan
Evidently is "an open-source Python library with over 40+ million downloads. It provides 100+ evaluation metrics, a declarative testing API, and a lightweight visual interface" per Evidently docs.
pip install evidently into the CI or monitoring environment (Step 1).Dataset with the same DataDefinition (Step 2).Report([DataDriftPreset()]) with
run(reference_data=, current_data=) and save_html for review (Step 3).include_tests=True, read .dict()["tests"], and
raise SystemExit on any FAIL/ERROR (Step 4).BinaryClassification
with target, prediction_probas and pos_label) and set the business
probas_threshold before trusting precision or recall (Step 5).psi, etc.) and severity tiers so
critical drift blocks and minor drift only alerts (Step 4).pip install evidentlySee the canonical install page at
https://docs.evidentlyai.com/docs/setup/installation for the current
install options (pip install evidently, plus the evidently[llm] extra).
The standard pattern compares two datasets:
import pandas as pd
reference_df = pd.read_parquet("reference.parquet")
current_df = pd.read_parquet("current.parquet")Without a data definition Evidently guesses each column's role, and two guesses cause most false or missing alerts (Evidently data definition):
categorical_columns.id is taken as the ID, and only one named
timestamp as the timestamp; both are ignored in drift. A sequential
account_number or order_id under any other name is checked as a feature
and drifts every day by construction, because new IDs are always larger.
Declare it as id_column (or drop it).from evidently import Dataset, DataDefinition
definition = DataDefinition(
id_column="account_number",
timestamp="event_time",
numerical_columns=["order_value", "items", "model_score"],
categorical_columns=["store_code", "channel"],
)
reference = Dataset.from_pandas(reference_df, data_definition=definition)
current = Dataset.from_pandas(current_df, data_definition=definition)Use the same definition for both datasets and pass the Dataset objects to
report.run(reference_data=reference, current_data=current).
from evidently import Report
from evidently.presets import DataDriftPreset
# The current API takes the preset list positionally; run() with keyword
# args is unambiguous about which dataset is which (per [Evidently Report]).
report = Report([DataDriftPreset()])
my_eval = report.run(reference_data=reference_df, current_data=current_df)
my_eval.save_html("drift_report.html")Result: HTML dashboard + structured JSON. Per Evidently docs, the preset bundles per-feature drift detection with sane defaults.
In the current Evidently API there is no separate TestSuite class. You
enable per-column pass/fail tests by passing include_tests=True to the
Report, then read each test's status from the result, per
Evidently Report:
from evidently import Report
from evidently.presets import DataDriftPreset
# include_tests=True turns the preset's per-column drift metrics into
# pass/fail tests alongside the metrics.
report = Report([DataDriftPreset()], include_tests=True)
my_eval = report.run(reference_data=reference_df, current_data=current_df)
# .dict() exposes top-level "metrics" and "tests" only - there is NO
# top-level "status" key. Gate on any test that did not pass.
result = my_eval.dict()
failed = [t for t in result["tests"] if t.get("status") in ("FAIL", "ERROR")]
if failed:
raise SystemExit(
f"Evidently drift gate failed: {len(failed)} test(s); see drift_report.html"
)Evidently's drift detection supports several statistical methods (psi,
wasserstein, ks, chisquare, jensenshannon); PSI is conventional for
tabular production drift. Configure the method and threshold per column on
the preset or the dataset's data definition, per Evidently drift preset.
Performance metrics need the task mapped in the data definition. Evidently
otherwise looks for its default names: target="target", predicted
probabilities in prediction, and pos_label=1 (Evidently data
definition). With string labels such as "yes"/"no" and a probability
column under another name, the default mapping does not measure the class
you care about.
from evidently import BinaryClassification, DataDefinition, Dataset, Report
from evidently.metrics import Recall
from evidently.presets import ClassificationPreset
from evidently.tests import gte
definition = DataDefinition(
classification=[BinaryClassification(
target="churned", # "yes" / "no"
prediction_probas="churn_score", # model probability, 0-1
pos_label="yes", # recall of the class the business acts on
)],
categorical_columns=["churned"],
)
cur = Dataset.from_pandas(cur_df, data_definition=definition)
ref = Dataset.from_pandas(ref_df, data_definition=definition)
# probas_threshold = the business decision threshold, not the 0.5 default
report = Report([
ClassificationPreset(probas_threshold=0.4),
Recall(probas_threshold=0.4, tests=[gte(0.7)]),
])
result = report.run(reference_data=ref, current_data=cur)ClassificationPreset, Recall and the other classification metrics take
probas_threshold, and any metric accepts test conditions such as gte
(Evidently all metrics, Evidently tests). Gate on the result as in
Step 4. Use RegressionPreset with a Regression(...) mapping for
regression models.
Labels often arrive late (chargebacks, returns, churn). Evaluate performance on a window whose labels have matured, and watch prediction drift for the most recent traffic in the meantime.
# Daily monitoring job
import datetime
from pathlib import Path
today = datetime.date.today().isoformat()
current_df = load_production_window(start=today, days=1)
reference_df = load_reference_window()
report = Report([DataDriftPreset()], include_tests=True)
result = report.run(reference_data=reference_df, current_data=current_df)
result.save_html(Path(f"monitoring/{today}.html"))
if any(t.get("status") in ("FAIL", "ERROR") for t in result.dict()["tests"]):
notify_oncall(f"Data drift detected on {today}")Pair with a scheduler (Airflow / Prefect / cron / Argo Workflows).
When the Step 6 job pages, triage the alert before acting - never jump straight to retrain or rollback.
Parse the alert envelope. Read the JSON report (report.dict() returns the
run as a dictionary per
Evidently output formats).
Each column is scored by a drift method against a threshold; defaults are PSI
and Jensen-Shannon divergence at threshold 0.1, KS and chi-square at p-value
0.05, per
Evidently customization docs.
Dataset-level drift triggers when the share of drifted columns reaches
drift_share: "By default, Dataset Drift is detected if at least 50% of
columns drift" per Evidently drift preset. Note which columns drifted and
which stat test fired.
Classify the drift signal:
| Signal | Look for |
|---|---|
| Broad feature drift (many columns) | Schema/ETL change or population shift |
| Single-column drift, especially an ID or timestamp | Pipeline bug or upstream encoding change |
| Target/prediction drift without feature drift | Concept drift or label-pipeline failure |
| Drift that aligns with calendar (weekend, holiday, season) | Seasonality - not a model failure |
| Drift only in serving data, not in a held-out eval set | Training-serving skew |
Rank root-cause hypotheses (default likelihood order in practice; adjust on the signal evidence) and act per hypothesis:
model-risk-evidence-matrix on the retrained candidate (fairness metrics
shift with population); update the reference dataset only after successful
promotion.Triage discipline:
A team ships a weekly retrain of a tabular fraud classifier and wants CI to block a release whose input distribution has moved too far from the validated baseline.
reference.parquet holds the last validated production week (~400k rows);
current.parquet holds the candidate model's eval slice.from evidently import Report
from evidently.presets import DataDriftPreset
report = Report([DataDriftPreset()], include_tests=True)
result = report.run(reference_data=reference_df, current_data=current_df)
result.save_html("drift_report.html")
failed = [t for t in result.dict()["tests"] if t.get("status") in ("FAIL", "ERROR")]
if failed:
raise SystemExit(f"Evidently drift gate failed: {len(failed)} test(s)")transaction_amount, merchant_country) drift past their PSI
threshold, so their tests report FAIL.raise SystemExit fails the CI job; drift_report.html is uploaded as a
build artifact showing the two drifted distributions.merchant_country gained a new region, widens that
column's threshold (or retrains on data that includes it), and the re-run
passes.| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Use yesterday as reference (rolling window only) | Slow drifts go undetected (model degrades 1% per day for 100 days = 100% drift) | Pin a stable reference (Step 2) |
| Run only on training data | Training data is curated; never reflects real production distribution | Use real production samples (Step 6) |
| Default thresholds for all metrics | Defaults are textbook; production tolerance differs | Tune per-feature thresholds (Step 4) |
| Block deploy on every drift | High-traffic production shifts daily; team disables monitor | Severity tiers: critical drift blocks; minor drift alerts |
| Integer codes or IDs left to type inference | Codes get numeric drift tests; a sequential ID drifts every day | Declare categorical_columns and id_column (Step 2) |
| Skip target/prediction drift | Concept drift (inputs stable, output behavior changed) goes undetected | Include the target/prediction column in the drift check (Steps 3-4) |
fairlearn-fairness skill.include_tests, run(), and
the .dict() result shape (top-level metrics + tests, per-test status)id_column,
categorical_columns and BinaryClassification mappingprobas_threshold and
test conditions such as gte