Monitoring machine learning models in production generates no shortage of metrics. Prediction latency, throughput, accuracy estimates, feature distributions, error rates—the instrumentation can track dozens of signals. The challenge isn’t collecting metrics; it’s identifying which ones reliably indicate problems that need intervention versus normal fluctuation that can be ignored.

Different types of ML applications have different critical metrics, but some patterns hold broadly. For classification models, tracking the prediction distribution over time catches a lot of issues. If your model normally predicts class A 60% of the time and class B 40%, and that shifts to 80/20, something has likely changed in your input data distribution or model behavior.

This doesn’t tell you what changed or whether it’s a problem, but it’s a reliable signal that something is different. You can then investigate whether it’s a genuine shift in the underlying phenomenon you’re modeling (fine, the model is responding to real changes) or data pipeline issues causing distribution shift (problem that needs fixing).

Feature distribution monitoring catches upstream data problems before they affect predictions. If a feature that normally ranges from 0-100 suddenly has values in the thousands, you probably have a data pipeline bug or source data change. Monitoring for outliers, missing values, and distribution changes in input features provides early warning of many issues.

The challenge with feature monitoring is scale. Models with hundreds of features generate too many individual metrics to monitor manually. Automated alerting on statistical changes (using techniques like KL divergence or Kolmogorov-Smirnov tests) helps, but tuning alert thresholds to avoid false positives while catching real issues requires iteration.

Prediction latency is critical for real-time serving systems but less informative for what’s wrong when it degrades. Latency can increase due to infrastructure issues (high load, resource constraints), model issues (inefficient inference code, model size), or upstream data issues (slow feature computation). Monitoring latency tells you there’s a problem, but not where.

That’s still useful—latency degradation often affects user experience before other metrics show issues. But you need additional monitoring to diagnose root cause. Tracking latency broken down by components (feature retrieval, model inference, post-processing) helps isolate where slowdowns occur.

Model accuracy or performance metrics are what most people instinctively want to monitor—is the model still accurate? But this is tricky in production because you often don’t have immediate ground truth labels. For a fraud detection model, you might not know if a transaction was actually fraudulent for days or weeks. Monitoring accuracy in real-time isn’t possible without labels.

What you can monitor is proxy metrics that correlate with accuracy. For models that affect user actions, conversion rates or user engagement can signal model degradation. If your recommendation model’s click-through rate drops, the model is likely performing worse even if you don’t have explicit accuracy measurements.

Some systems implement online evaluation using a small sample of production traffic where ground truth is obtained quickly. For example, showing recommended items to users and tracking immediate engagement. This provides ongoing accuracy estimates on a subset of traffic, giving directional signals about model performance.

Error rate monitoring catches obvious failures—predictions that error out rather than returning a result. This should be a basic production metric for any system, not ML-specific. But it’s worth breaking down by error type. Errors from missing features indicate data pipeline issues. Errors from invalid inputs indicate upstream changes. Errors from model code indicate model or serving infrastructure issues.

Concept drift is the ML-specific phenomenon where the relationship between features and predictions changes over time. A model trained on historical data may become less accurate as the world changes. Detecting this requires either periodic retraining and comparing new model performance to production model performance, or tracking proxy metrics that suggest drift.

Monitoring prediction confidence can provide drift signals. If a model’s average confidence score decreases over time, it might indicate the model is encountering patterns it’s less certain about, suggesting the data distribution has shifted from training data. This isn’t definitive—confidence calibration varies—but directional changes are informative.

Data quality metrics often predict model issues before performance degrades noticeably. Missing values, unexpected nulls, format changes, schema violations—these indicate upstream problems that will eventually affect model accuracy. Catching them early lets you address root causes before they impact predictions.

One underused metric is prediction diversity or entropy. For recommendation systems or ranking models, if predictions become increasingly similar across users or contexts, it might indicate the model is overfitting to a dominant pattern or has stopped adapting to individual contexts. Monitoring the distribution of predictions catches this.

The frequency of monitoring matters too. Latency and throughput should be monitored continuously with second-to-minute granularity—these change quickly and affect user experience immediately. Feature distributions and prediction distributions can be monitored hourly or daily—they change more slowly. Accuracy proxies depend on label availability but often daily or weekly is sufficient.

Alert fatigue is real. If your monitoring system generates alerts constantly for normal variations, people start ignoring them and miss actual issues. Tuning alert thresholds to signal actual problems, not just statistical deviations from baseline, requires ongoing adjustment based on operational experience.

I’ve found that combining multiple signals works better than relying on any single metric. A slight change in prediction distribution alone might not be concerning. That same change combined with increased latency and decreased confidence scores suggests a real problem worth investigating.

The other critical aspect is having runbooks for common alerts. When prediction latency spikes, what should you check first? When feature distributions shift, what’s the investigation process? Documented response procedures turn monitoring alerts into actionable workflows rather than vague warnings.

For teams operating multiple models, centralizing monitoring helps identify systemic issues versus model-specific ones. If latency increases across all models simultaneously, it’s probably infrastructure. Team400 has built centralized monitoring dashboards that aggregate metrics across model families effectively. If one model’s prediction distribution shifts while others are stable, it’s likely model or data-specific.

One pattern I’ve seen work well is layered monitoring: infrastructure metrics (CPU, memory, request rate), system metrics (latency, throughput, error rate), and ML-specific metrics (feature distributions, prediction distributions, accuracy proxies). Issues usually show up in multiple layers, and the combination points to root cause more effectively than any single layer.

The return on investment in good monitoring is substantial. Catching data pipeline issues before they corrupt training data prevents deploying bad models. Detecting model performance degradation early enables timely retraining. Identifying infrastructure bottlenecks prevents service degradation.

But monitoring effectiveness depends on knowing what signals matter for your specific system. The metrics that predict issues for a real-time fraud detection system differ from those relevant to a batch recommendation system. Generic MLOps advice provides starting points—operational experience refines it to what actually works for your models and use cases.