Machine learning models decay. A model trained on last year’s data and performing well in production might gradually or suddenly start performing worse as the real-world data distribution changes. This is model drift, and managing it is one of the core challenges of ML operations.

There are two main types of drift. Covariate shift is when the distribution of input features changes but the relationship between inputs and outputs stays the same. Concept drift is when the actual relationship changes—what used to predict the target variable no longer does.

These require different responses. Covariate shift often means you need to retrain on more recent data that reflects the new input distribution. Concept drift might mean you need to revisit feature engineering or even your model architecture.

The challenge is detecting drift before it significantly impacts performance. Waiting until user complaints roll in means the damage is already done. Proactive monitoring catches drift early.

Feature distribution monitoring is the most straightforward approach. Track summary statistics of your input features over time. Mean, median, standard deviation, percentiles. If these drift significantly from the training distribution, something has changed.

The question is what constitutes “significant.” Small fluctuations are normal. You need thresholds that capture meaningful drift without triggering false alarms on random variation. This requires understanding your specific domain and feature stability.

For some features, any change is suspicious. A categorical feature that should have five values suddenly showing a sixth value indicates a data pipeline change or a new scenario you didn’t anticipate. For continuous features, you need statistical tests.

The Kolmogorov-Smirnov test compares distributions and can detect when the production feature distribution diverges from the training distribution. The Population Stability Index (PSI) is commonly used in credit scoring and quantifies distribution shift in a single number.

These tests give you quantitative drift metrics, but you still need to choose thresholds that trigger alerts. Set them too sensitive and you get alert fatigue. Set them too loose and you miss important drift.

Prediction distribution monitoring watches the model’s outputs. If your model usually outputs predictions distributed in a certain way, and suddenly the distribution shifts, that’s a signal. Maybe your model is now predicting the positive class 50% of the time when historically it was 10%. Something changed.

This doesn’t directly tell you if the model is right or wrong—maybe the true positive rate actually did increase. But it tells you that the world the model is seeing looks different. You should investigate.

Performance monitoring is essential but has a lag problem. You can’t measure performance until you get ground truth labels, which might be days, weeks, or months after predictions. A fraud detection model makes predictions instantly but you might not know if fraud actually occurred until much later.

For problems with fast ground truth feedback, performance monitoring is straightforward. Track accuracy, precision, recall, or whatever metrics matter for your use case. When they drop below acceptable thresholds, investigate.

Shadow scoring is a technique for problems with delayed ground truth. Deploy a new model alongside your production model, send both the same inputs, but only serve predictions from the production model. Compare their predictions. If they diverge significantly, one of them is likely experiencing drift (or the new model has different behavior you need to understand).

This requires running two models in production, which doubles compute costs for inference. But it provides early warning of drift before ground truth arrives.

Not all drift requires immediate retraining. Sometimes drift indicates a data quality issue rather than a genuine distribution change. A sudden spike in missing values for a feature might mean upstream data collection broke. Retraining on this data would be counterproductive.

Investigation comes before reaction. When drift is detected, first understand what changed and why. Check data pipelines. Review recent changes to upstream systems. Talk to stakeholders about known business changes. Sometimes drift has an obvious explanation that informs your response.

Temporary drift might not warrant retraining. If a feature distribution shifts for a week due to a holiday or special event, then returns to normal, retraining might not be necessary. The model might perform adequately through the temporary shift.

Seasonal patterns complicate drift detection. Features that vary seasonally will show drift if you compare to the wrong baseline. You need to compare current December data to last December, not to last June. Otherwise you’re detecting seasonal patterns, not drift.

This requires maintaining historical distributions segmented appropriately. For seasonal domains, you need at least a year of history to establish expected variation patterns.

Retraining frequency is a balancing act. Too frequent and you’re wasting compute and risking instability from models that haven’t converged on stable patterns. Too infrequent and you’re serving degraded predictions.

Some systems retrain on a schedule—weekly, monthly, quarterly. This works if drift is gradual and predictable. For domains with unpredictable shifts, event-triggered retraining based on drift detection makes more sense.

Feature importance shifts indicate concept drift. If a feature that was previously important becomes uninformative, or vice versa, the underlying relationship between features and target has changed. This is harder to detect than distribution shifts but more concerning for model validity.

SHAP values or other explainability metrics tracked over time can reveal these shifts. If the average SHAP value for a feature changes significantly, investigate why.

A/B testing new models against production models provides a gold standard for evaluating whether retraining actually helps. Deploy the retrained model to a small percentage of traffic. Compare performance to the existing model on the same data. If the new model isn’t actually better, don’t fully deploy it.

This catches cases where drift exists but retraining doesn’t solve it, possibly because you need feature engineering changes or the drift is in a direction your model architecture can’t adapt to.

Automated retraining pipelines are becoming standard in mature ML operations. Drift triggers a retraining job, new model is validated on holdout data, if validation metrics exceed thresholds the model is promoted to shadow scoring, if shadow scoring shows improvement it’s promoted to production.

This requires robust infrastructure and confidence in your validation process. Fully automated retraining without human oversight is risky unless you have extensive safeguards.

Documentation of drift incidents builds institutional knowledge. When drift occurred, what caused it, how it was detected, what the response was, and what the outcome was. This history informs future drift detection threshold tuning and response protocols.

Some domains experience minimal drift and models can run for years with minor degradation. Other domains drift constantly and need frequent retraining. Understanding your domain’s drift characteristics prevents both over-investment in drift management for stable domains and under-investment for volatile ones.

External validity monitoring compares your model’s performance to simple baselines. If a naive model predicting the most common class outperforms your complex ML model, something is very wrong. This catches catastrophic drift that might slip through other monitoring.

User feedback, when available, provides qualitative drift signals. If users suddenly report that recommendations seem irrelevant, or predictions seem wrong, even if your metrics look fine, take it seriously. Users are experiencing something your monitoring might not capture.

The cost of drift-related errors varies by application. A recommendation system showing slightly less relevant products is annoying but not catastrophic. A medical diagnosis model becoming less accurate is potentially dangerous. Your drift response urgency should match the stakes.

For critical applications, conservative thresholds and rapid response protocols make sense even if they result in more frequent retraining than strictly necessary. For low-stakes applications, you can tolerate more drift and respond more slowly.

Drift detection is an ongoing process, not a one-time setup. As you learn more about your domain’s drift patterns, you refine thresholds, add new monitoring metrics, and adjust response protocols. The monitoring infrastructure evolves alongside your understanding.

The reality is that all production ML models drift eventually. The question is whether you detect it proactively through monitoring or reactively through degraded outcomes. Investing in drift detection infrastructure pays for itself by catching problems before they impact users.