When to run it
Falling business metrics, a drift alert, or a model that has not been retrained since it shipped.
The model is drifting.Retrain it behind a gate.
Catching model drift and retraining safely
The playbook in one paragraph
A production model quietly gets worse as the world changes. Ana detects the drift, diagnoses which features and slices moved, retrains on fresh data, and promotes the new model only if it beats the old one through shadow and canary.
Works with
- Feature store or warehouse (Snowflake, BigQuery, Databricks)
- Training pipelines (SageMaker, Vertex AI, Kubeflow, MLflow)
- Model registry
- Serving and monitoring
What Ana needs from you
- Evaluation metrics and slice definitions that matter
- Promotion thresholds
- Data windows allowed for retraining
How Ana runs it
Step by step.
5 steps from trigger to done - each one visible to your team as it happens.
- 01
Detect and localize
Ana monitors input and prediction distributions and pinpoints which features, segments and time windows shifted.
- 02
Validate the new data
Before training, Ana checks schema, freshness, label quality and leakage in the new window, and stops if the data is not fit to train on.
- 03
Retrain and evaluate
Ana retrains with your pipeline and evaluates on accuracy, latency and every defined slice against the current production model.
- 04
Shadow, then canary
The candidate scores live traffic in shadow, then serves a small canary with automatic rollback if any gate fails.
- 05
Promote and record
On approval the model is promoted in the registry with a model card, evaluation report and lineage.
Who approves what
Approval gates
- Training on new data window
- ML lead
- Canary start
- Model owner
- Full promotion
- Model owner, after gates pass
What you get
Deliverables
- 01Drift diagnosis
- 02Data validation report
- 03Evaluation against production on every slice
- 04Promoted model with model card and lineage
What usually goes wrong
And how Ana handles it.
Risk
A better average that is worse for a key segment
Slice-level gates block promotion even when the overall metric improves.
Risk
Training on bad data
Data validation is a hard gate before any training run.
FAQ
Common questions.
No. It runs on your existing pipelines, registry and serving stack.
Keep going
Related playbooks.
AI / ML
Taking a notebook model to production
A promising model lives in a data scientist’s notebook. Ana turns it into a reproducible pipeline, a tested serving endpoint and a monitored deployment - without rewriting the science.
Read the playbook →Software
Upgrading a framework across dozens of services
A major framework or runtime version is going end-of-life and it touches every service. Ana plans the migration, upgrades each service in its own branch, fixes what breaks, and hands your team small, tested pull requests instead of one terrifying diff.
Read the playbook →Software
Burning down a stale bug backlog
Every team has a backlog of real but low-priority bugs that never win a sprint. Ana triages them, reproduces the ones that are still real, fixes them with tests, and closes the rest with evidence.
Read the playbook →See it in your stack
Run this playbook on your systems.
Thirty minutes, your repositories and cloud, one real instance of this job.