MLOps in Production: What It Takes to Keep a Predictive Model Running

Keeping a predictive model running in production takes seven controls beyond the model itself: versioned feature pipelines, protection against training-serving skew, drift monitoring, defined retraining triggers, a model registry, approval with tested rollback, and a named owner. Without them, accuracy decays quietly while the dashboard still looks healthy.

Most organisations can now train a good model. Far fewer can say, eighteen months later, whether that model still deserves the decisions it is making. The gap between those two states is what MLOps exists to close, and it is mostly engineering and ownership rather than data science.

#Why Predictive Models Decay After Go-Live

What this means for you: a model that passed validation on launch day is a statement about the past; production is where it meets a world that keeps changing.

In 2015 a team of Google engineers led by D. Sculley published "Hidden Technical Debt in Machine Learning Systems" at NeurIPS. Its best-known figure shows the machine learning code as a small box in the middle of a much larger system: data collection, feature extraction, configuration, serving infrastructure and monitoring. The paper named risks that still catch teams out, including entanglement between features, hidden feedback loops and undeclared consumers of a model's output.

Nobody switches a predictive model off. It goes on producing confident numbers long after it has stopped being right.

The reason is simple. A model learns relationships from historical data, and the relationships move. Customer behaviour shifts with prices and seasons, upstream systems change a field's meaning, and a new product line appears that the training data never saw. None of these raise an error. The model keeps scoring, and the scores keep looking plausible.

That silence is what separates model failure from ordinary software failure. A broken API returns an error code. A decaying model returns a number, and the cost appears months later as wrong credit decisions, excess stock or missed fraud.

#What MLOps Actually Covers

What this means for you: if your definition of MLOps stops at automated deployment, you have built the easy half.

MLOps, short for machine learning operations, is the set of engineering practices that takes a model from experiment to production and keeps it trustworthy there. It borrows from DevOps the ideas of version control, automated testing and continuous delivery, then adds what software does not need: versioned data, reproducible training, monitoring of statistical behaviour, and governed retraining.

It is distinct from two neighbours. Data observability watches the health of datasets and pipelines; our guide to data observability and quality management covers that layer, and MLOps depends on it. Model governance is the control framework a regulator or risk committee expects; MLOps produces the evidence that framework needs.

Google Cloud's widely cited MLOps guidance describes three maturity levels. Level 0 is a manual process, with notebooks and hand-run training. Level 1 automates the training pipeline so models retrain on new data. Level 2 adds continuous integration and delivery for the pipelines themselves. The table below extends that view with the controls that decide whether a model can be trusted, not only redeployed.

#Feature Pipelines and Training-Serving Skew

What this means for you: most production model failures we are asked to diagnose trace back to features, not to the algorithm.

Training-serving skew is the gap between the data a model learned from and the data it receives in production. It arises when features are computed twice: once in a batch pipeline for training, and again in application code for real-time scoring. The two implementations drift apart through a different time zone, a different null rule, or a join that uses yesterday's reference data. Accuracy falls, and every test still passes.

Feature pipelines are data engineering, and they deserve the same discipline as any production pipeline: tests, version control and contracts with upstream producers. A data contract, an agreed and enforced specification of a dataset's schema and meaning, stops a source team from silently changing a field a model depends on. That work sits naturally inside a wider data engineering and integration practice rather than in a separate data science silo.

#Point-in-Time Correctness

A second source of skew is leakage: training on information that would not have been available at the moment of prediction. A churn model trained on data that includes the cancellation date's account state will look excellent in validation and fail in production. Point-in-time joins, which reconstruct each feature as it stood at the prediction timestamp, are the defence, and they need to be built into the pipeline rather than remembered by each analyst.

#Drift Monitoring That Catches Decay Early

What this means for you: if the first person to notice a model has degraded is a customer or a finance director, your monitoring is measuring the wrong things.

There are three things worth watching, and they arrive at different speeds. Input drift, a change in the distribution of the features, is visible immediately. Prediction drift, a change in the distribution of the model's outputs, is also immediate. Outcome drift, a fall in real accuracy, is the one that matters most, and it arrives last, because ground truth often takes weeks or months to land.

Because outcome data lags, mature teams use input and prediction drift as early warnings and outcome metrics as the confirmation. The thresholds should be set per model, against business impact, rather than copied from a tool's defaults. Alert fatigue is the most common reason drift monitoring gets switched off, and the same principles that keep data quality alerts meaningful apply here.

#Retraining Triggers, Registry and Rollback

What this means for you: retraining is a release, and it needs the same control as any other change to a system that makes decisions.

#When to Retrain

There are three common triggers: a schedule, a drift threshold and an outcome threshold. Scheduled retraining is simple and predictable but wastes compute when nothing has changed and can be too slow when something has. Drift-triggered retraining responds faster but can chase noise. Outcome-triggered retraining is the most defensible, limited by how quickly ground truth arrives. Many estates combine a schedule as a floor with outcome thresholds as an accelerator.

Automatic retraining is not the same as automatic release. A retrained model should be evaluated against the current champion on a held-out recent window, and promoted only if it is better on the metrics the business cares about.

#The Registry as the System of Record

A model registry stores every version of a model with its metadata: the code commit, the training data snapshot, the feature definitions, the evaluation results and who approved it. That record, together with data lineage back to source systems, answers the question every incident and every audit eventually asks: which model made this decision, trained on what, and approved by whom?

The registry is where a model stops being a data scientist's artefact and becomes an asset the organisation can account for.

#Approval and Rollback

Promotion to production should pass a gate: automated tests on data and performance, then human approval proportionate to the model's impact. Rollback should be a tested, one-step operation to the previous registered version. If rolling back requires retraining, there is no rollback. Treating these gates as code, versioned and enforced in the pipeline, follows the same pattern as governance enforced in CI/CD.

#Ownership, Regulation and Cost

What this means for you: an alert without an owner is a log entry; decide who acts before the first one fires.

Every production model needs a named business owner, accountable for whether its decisions are acceptable, and a named technical owner, accountable for keeping it running. Without both, drift alerts land in a shared inbox and wait. The operating model that makes this sustainable is the same platform-team pattern described in our guide to platform engineering and DataOps.

Regulation is raising the floor in some sectors. The Prudential Regulation Authority's supervisory statement SS1/23 on model risk management took effect on 17 May 2024 for UK banks with internal-model approval for regulatory capital, and it explicitly covers risks from machine learning techniques within its general model framework. Under the EU AI Act, obligations for high-risk systems were deferred to 2 December 2027 by the Digital Omnibus amendments adopted in 2026, and they include post-market monitoring. Teams building generative systems will find the evidence model overlaps with our GenAI governance operating model.

Cost needs the same attention. Retraining, feature computation and real-time serving all consume compute, and serving at low latency is usually the largest line. Tag every model's training and serving spend, and review it against the value the model is meant to protect. Where the platform is still being chosen, our Snowflake versus Databricks decision guide covers how each handles machine learning workloads.

#When Full MLOps Is Overkill

What this means for you: match the investment to the model's impact and change rate, not to the maturity model's top level.

Full Level 2 automation is the wrong investment for a handful of low-impact models that change slowly. A quarterly demand model used to guide, not dictate, a planning conversation can be retrained by hand with a checklist, a registry entry and a basic drift report. Building continuous delivery for it costs more than any error it could cause.

It is also premature when the data underneath is not stable. If source systems are mid-migration, feature definitions are contested, or there is no agreed ground truth, automation will only make inconsistent models faster. Fix the data foundation first; an AI readiness assessment is often the cheaper first step. And for language-model applications that call external models rather than training their own, the controls look different; our guide to production agentic AI architecture covers that ground.

The proportionate rule: invest in automation where models are high-impact, change quickly or are numerous, and keep a disciplined manual process everywhere else.

#Where to Start: The First Ninety Days

What this means for you: start with an inventory and one model, not with a platform purchase.

The scope is deliberately narrow: ninety days brings one high-impact model up to standard and leaves a reusable pattern, not an estate-wide platform. Extending it across a portfolio of models is the multi-quarter work, best run as a shared capability with machine learning engineering for predictive models and CI/CD and DevOps for data platforms working from the same pipeline templates.

Write down what you find in the inventory. It is usually the most useful document the programme produces, because it shows the board how many decisions already depend on models nobody is watching.

#Frequently Asked Questions

#What is MLOps and why does it matter?

MLOps, short for machine learning operations, is the set of engineering practices that takes a model from experiment to production and keeps it trustworthy there. It covers versioned feature pipelines, monitoring for drift, governed retraining, a model registry, approval gates and rollback. It matters because predictive models decay silently as the world changes, while still producing plausible-looking numbers.

#What is training-serving skew?

Training-serving skew is a gap between the data a model learned from and the data it receives in production. It usually happens when features are computed twice, once in a batch pipeline for training and again in application code for scoring, and the two drift apart. The structural fix is a single, versioned feature definition used for both training and serving.

#How often should a production model be retrained?

There is no single right interval. Common triggers are a fixed schedule, a drift threshold on inputs or predictions, and a fall in measured accuracy against real outcomes. Many teams use a schedule as a minimum and outcome thresholds to retrain sooner. Every retrained model should be compared with the current one on recent data before it is promoted.

#What should a model registry record?

A model registry should record every model version with the code commit, the training data snapshot, the feature definitions, the evaluation results, the approver and the deployment history. Together with data lineage, that lets a team answer which model made a given decision, what it was trained on and who approved it, which incident reviews and regulators both ask.

#When is full MLOps automation not worth it?

Full automation is rarely worth it for a few low-impact models that change slowly, where a manual retraining checklist, a registry entry and a basic drift report are proportionate. It is also premature when source data is unstable or ground truth is undefined. Invest in automation where models are high-impact, change quickly or are numerous.

Unolabs is a Data and AI first engineering consultancy, headquartered in the United Kingdom with engineering operations in Pune and active engagements across the UK, Australia, and Hong Kong. We help enterprises build the architectural foundation for autonomous AI execution — governed data platforms, semantic intelligence, and agentic systems that enterprises can stand behind.

If you are weighing how to keep your predictive models trustworthy after go-live, book a discovery call and we will work through it with you.