MLOps for Data Engineers: MLflow Tracking, Model Registry, Deployment, and Drift Monitoring

Table of Contents

• Why MLOps Is a Data Engineering Problem
• The Machine Learning Lifecycle
• Why Models Fail in Production
• MLflow — The Standard Toolkit
• Experiment Tracking — Never Lose a Good Model Again
• Model Signatures and Input Examples
• The Model Registry in Unity Catalog
• Aliases — Champion, Challenger, and Instant Rollback
• Deployment Patterns — Batch, Streaming, and Real-Time
• Batch Inference with Spark
• Feature Stores and Training-Serving Skew
• Monitoring — Data Drift, Concept Drift, and Model Decay
• Measuring Drift with PSI
• Retraining Strategies
• CI/CD for Machine Learning
• MLOps Maturity Levels
• MLOps on Fabric and Azure Machine Learning
• Common Mistakes
• Interview Questions
• Wrapping Up

The earlier posts in this series taught you how to build models: regression, XGBoost, neural networks, evaluation, and tuning. But a model sitting in a notebook delivers zero business value. It has to be versioned, deployed, fed fresh data every day, monitored, and retrained when the world changes. That discipline is MLOps — and most of it is data engineering.

This post covers the full production lifecycle with MLflow, the open-source standard built into Databricks and Microsoft Fabric: experiment tracking, model signatures, the Unity Catalog model registry, champion/challenger promotion, batch and real-time deployment, feature stores, drift monitoring, retraining, and CI/CD. The MLflow code in this post was run end to end, and every drift number is calculated.

Analogy — A restaurant chain launching a new dish. A chef perfecting a recipe in a test kitchen is model development: lots of experiments, notes on every attempt, tasting and adjusting. But a chain serving that dish in 300 locations needs much more: a written recipe card everyone follows (the registered model), the same ingredient suppliers in every location (the feature pipeline), a way to roll out a new version to a few restaurants first (champion/challenger), food inspectors checking quality every week (monitoring), and a process for updating the recipe when customer tastes change (retraining). MLOps is everything between “the chef’s recipe works” and “300 restaurants serve it reliably every night.”

Why MLOps Is a Data Engineering Problem

What a data scientist typically owns:
  Choosing the algorithm, engineering features, tuning, evaluating

What a data engineer typically owns (or co-owns):
  Training data pipelines        Clean, versioned, reproducible datasets
  Feature pipelines              The same transformations in training AND inference
  Scheduled training jobs        Lakeflow Jobs / pipelines that retrain on new data
  Batch scoring                  Applying the model to millions of rows daily
  Prediction tables              Landing scores in Delta tables for BI and apps
  Monitoring tables              Input distributions, drift metrics, alerts
  Environments and CI/CD         Dev → staging → prod, permissions, deployments

A rough rule from practice: the model code is a small fraction of a
production ML system. The data collection, validation, feature pipelines,
serving infrastructure, and monitoring around it are the bulk of the work.

The Machine Learning Lifecycle

┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
   │ 1. Data  │ → │ 2. Train │ → │ 3. Track │ → │4.Register│
   │ prep and │   │  models  │   │  every   │   │ best one │
   │ features │   │          │   │   run    │   │          │
   └──────────┘   └──────────┘   └──────────┘   └────┬─────┘
        ▲                                            │
        │                                            ▼
   ┌──────────┐   ┌──────────┐                  ┌──────────┐
   │8. Retrain│ ← │7. Monitor│ ← ─ ─ ─ ─ ─ ─ ─  │ 5. Deploy│
   │ on fresh │   │drift and │   6. Predict     │ (batch / │
   │   data   │   │ accuracy │   every day      │real-time)│
   └──────────┘   └──────────┘                  └──────────┘

It is a loop, not a line. A model is never "done": the data it sees in
production keeps changing, so the cycle repeats for as long as the model
is in use.

Why Models Fail in Production

1. TRAINING-SERVING SKEW
   The model was trained on features computed one way (a pandas notebook)
   and served features computed another way (a SQL view written later).
   Same column name, different numbers. The model silently degrades.

2. DATA DRIFT
   Customer behavior, prices, or sensors change. A fraud model trained on
   pre-holiday transactions sees holiday spending patterns it never learned.

3. UPSTREAM PIPELINE CHANGES
   A source system starts sending revenue in cents instead of dollars, or
   a column fills with nulls after a schema change. The model keeps
   predicting -- confidently and wrongly.

4. NO REPRODUCIBILITY
   "Which data and settings produced the model in production?" Nobody
   knows. It cannot be rebuilt, audited, or debugged.

5. MANUAL DEPLOYMENTS
   A pickle file copied to a server by hand. No version history, no
   rollback, no approval trail.

6. NO MONITORING
   Accuracy is only discovered to have dropped months later, when the
   business notices bad decisions.

Notice that most of these are data and pipeline problems, not algorithm problems. That is why MLOps sits so naturally with data engineering.

MLflow — The Standard Toolkit

MLflow is an open-source platform for managing the ML lifecycle. It comes pre-installed and fully managed in Databricks and Microsoft Fabric, and runs anywhere else with a self-hosted server.

MLflow components:

  Tracking        Log parameters, metrics, artifacts, and code version for
                  every training run; compare runs in a UI
  Models          A standard package format ("flavors" for scikit-learn,
                  XGBoost, PyTorch, Spark ML, LLMs...) plus a universal
                  pyfunc interface: load any model, call .predict()
  Model Registry  Versioned, governed catalog of models with aliases and
                  permissions (Unity Catalog on Databricks)
  Evaluation and  MLflow 3 adds GenAI evaluation, tracing, and a prompt
  GenAI tools     registry for LLM applications and agents

Key vocabulary:
  Experiment   A folder of related runs ("churn-model")
  Run          One training attempt, with its params, metrics, artifacts
  Artifact     Any file a run produces: the model, plots, data samples
  Model version  An immutable snapshot of a registered model
  Alias        A movable name pointing at one version ("champion")

Analogy — A lab notebook, a packaging standard, and a pharmacy shelf. Tracking is the scientist’s lab notebook: every experiment recorded with exact conditions and results. The Models format is a standard medicine package with ingredients listed on the label, so any pharmacy can dispense it. The registry is the pharmacy shelf: only approved, labeled, versioned products go there, and the pharmacist always knows which batch is current.

Experiment Tracking — Never Lose a Good Model Again

Without tracking, experiments live in notebook cell outputs and memory. “The model from Tuesday was better” is impossible to verify or reproduce.

import mlflow
import pandas as pd
from mlflow.models import infer_signature
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score, f1_score

# On Databricks or Fabric, tracking is configured for you.
mlflow.set_experiment("/Shared/churn-model")

# Synthetic churn data: 5,000 customers, ~20% churners
X, y = make_classification(n_samples=5000, n_features=20, n_informative=10,
                           weights=[0.8, 0.2], random_state=42)
X = pd.DataFrame(X, columns=[f"feature_{i}" for i in range(20)])
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

for n_estimators, max_depth in [(100, 5), (200, 10), (300, None)]:
    with mlflow.start_run(run_name=f"rf_{n_estimators}_{max_depth}"):
        params = {"n_estimators": n_estimators, "max_depth": max_depth, "random_state": 42}
        model = RandomForestClassifier(**params).fit(X_train, y_train)

        proba = model.predict_proba(X_test)[:, 1]
        mlflow.log_params(params)
        mlflow.log_metrics({
            "test_auc": roc_auc_score(y_test, proba),
            "test_f1": f1_score(y_test, (proba > 0.5).astype(int)),
        })

        signature = infer_signature(X_train, model.predict(X_train))
        mlflow.sklearn.log_model(
            model,
            name="model",
            signature=signature,
            input_example=X_train.head(3),
            serialization_format="cloudpickle",
        )

Results from this run (three runs, compared in the MLflow UI or by code):

  n_estimators  max_depth  test_auc  test_f1
  ────────────  ─────────  ────────  ───────
  100           5          0.9363    0.6536
  200           10         0.9650    0.8291
  300           None       0.9656    0.8533   ← best

Notes on the code:
  log_params / log_metrics   Settings and results, searchable later
  log_model                  The trained model + environment + signature
  input_example              A few sample rows stored with the model
  serialization_format       Recent MLflow versions default to a stricter
                             format for scikit-learn models; setting
                             "cloudpickle" keeps this example working across
                             versions. Use it only for models you trained.

Pick the best run in code rather than by eye, so pipelines can automate it:

best = mlflow.search_runs(order_by=["metrics.test_auc DESC"], max_results=1)
best_run_id = best.iloc[0]["run_id"]
print(best.iloc[0][["params.n_estimators", "params.max_depth", "metrics.test_auc"]])

Autologging. mlflow.autolog() captures parameters, metrics, and models automatically for scikit-learn, XGBoost, LightGBM, PyTorch, Spark ML, and more. On Databricks it is enabled by default for many libraries. Explicit logging, as above, is still worth it for business metrics like F1 at your chosen threshold.

Model Signatures and Input Examples

A signature is the model’s contract: the exact input columns, their types, and the output shape.

Signature for the churn model:

  inputs:  [feature_0: double, feature_1: double, ... feature_19: double]
  outputs: [long]

Why it matters:
  Unity Catalog REQUIRES a signature to register a model
  Serving endpoints use it to validate incoming requests
  Batch scoring with spark_udf maps columns by name using it
  It documents the model's interface for everyone who uses it

Common failure:
  "Registration fails" or "the endpoint rejects my request"
  → missing signature, or column names/types that do not match

Analogy — An electrical plug standard. The signature is the shape of the plug. Any socket built to that standard (a batch job, an API, a dashboard) can use the model safely. Try to plug in the wrong shape and it fails immediately, instead of silently producing garbage.

The Model Registry in Unity Catalog

On Databricks, models are registered in Unity Catalog — the same governance layer as your tables. In MLflow 3 this is the default registry, and models use the familiar three-level name.

import mlflow
from mlflow import MlflowClient

mlflow.set_registry_uri("databricks-uc")   # Unity Catalog (default in MLflow 3 on Databricks)

model_name = "ml_prod.churn.churn_model"   # catalog.schema.model
version = mlflow.register_model(f"runs:/{best_run_id}/model", model_name)
print(f"Registered {model_name} version {version.version}")

What Unity Catalog adds:

  Three-level names      ml_dev.churn.churn_model vs ml_prod.churn.churn_model
                         -- separate catalogs per environment
  Permissions            EXECUTE on a model lets a user or job load it;
                         owners and MANAGE control changes -- the same
                         grant model as your tables
  Lineage                Which tables trained this model; which jobs use it
  Audit logs             Who registered or changed what, and when
  Versions               Every registration creates a new immutable version

Permissions you need to register:
  USE CATALOG on the catalog, USE SCHEMA and CREATE MODEL on the schema

Aliases — Champion, Challenger, and Instant Rollback

The legacy workspace registry used fixed stages (Staging, Production, Archived). Unity Catalog replaces them with aliases: named, movable pointers to a version. You can define any aliases you like, and production code loads a model by alias instead of by version number.

client = MlflowClient()

# Point "champion" at the version that should serve production traffic
client.set_registered_model_alias(model_name, "champion", version.version)

# Production code never hard-codes a version number
model = mlflow.pyfunc.load_model(f"models:/{model_name}@champion")
predictions = model.predict(X_test)

The champion / challenger pattern:

  Version 3 → @champion     serves production
  Version 4 → @challenger   newly trained, being evaluated

  Evaluate the challenger on fresh holdout data (or shadow traffic):
    challenger AUC 0.971 vs champion AUC 0.958 → promote

  Promotion = move one alias:
    set_registered_model_alias(name, "champion", 4)
    set_registered_model_alias(name, "previous", 3)   ← keep for rollback

  Rollback = move it back:
    set_registered_model_alias(name, "champion", 3)

  No code changes, no redeployment of jobs -- every consumer that loads
  "@champion" picks up the new version on its next run.

Stages vs aliases:
  Legacy stages          Unity Catalog aliases
  ─────────────          ─────────────────────
  Fixed names            Any names you choose
  One Production slot    Multiple aliases, side by side
  models:/name/Production   models:/catalog.schema.name@champion
  Deprecated             Recommended

Analogy — The “Today’s Special” sign. A restaurant keeps every recipe version in its binder (model versions). The chalkboard sign out front says which one is today’s special (the champion alias). Changing the special means rewriting the sign, not reprinting every menu, and if customers hate the new dish, you change the sign back by lunchtime.

Deployment Patterns — Batch, Streaming, and Real-Time

Start with batch whenever you can. It is the cheapest, simplest, and easiest to monitor. Most “we need real-time” requirements turn out to be satisfied by scores refreshed every hour or every night.

Batch Inference with Spark

mlflow.pyfunc.spark_udf turns any registered model into a Spark function, so you can score billions of rows in parallel across the cluster.

import mlflow
from pyspark.sql import functions as F

model_uri = "models:/ml_prod.churn.churn_model@champion"

# Wrap the model as a Spark UDF (works for any MLflow flavor)
predict = mlflow.pyfunc.spark_udf(spark, model_uri, result_type="double")

customers = spark.table("ml_prod.churn.customer_features")
feature_cols = [f"feature_{i}" for i in range(20)]

scored = (
    customers
    .withColumn("churn_prediction", predict(*[F.col(c) for c in feature_cols]))
    .withColumn("scored_at", F.current_timestamp())
    .withColumn("model_version", F.lit(model_uri))
)

(scored
    .select("customer_id", "churn_prediction", "scored_at", "model_version")
    .write.mode("append")
    .saveAsTable("ml_prod.churn.churn_predictions"))

Production habits for batch scoring:

  Append, do not overwrite    Keep prediction history; you will need it to
                              measure accuracy once outcomes are known
  Record the model version    Every prediction row says which model made it
  Load by alias               "@champion" -- promotion needs no code change
  Schedule as a job           Lakeflow Jobs / Fabric pipeline after the
                              feature tables refresh
  Validate inputs first       Null checks and range checks before scoring

Feature Stores and Training-Serving Skew

Training-serving skew happens when features are computed differently at training time and at prediction time. It is one of the most common causes of silent model failure.

How skew sneaks in:

  Training (data scientist notebook):
    avg_order_value = mean of ALL orders, including returns

  Serving (SQL view written months later by another team):
    avg_order_value = mean of COMPLETED orders only

  Same column name. Different numbers. The model's accuracy drops and
  nobody can see why.

Point-in-time correctness (the subtler version):
  When building training rows for "did the customer churn in March?",
  features must use only data available BEFORE March. Using a feature
  table's current values leaks future information into training and makes
  offline accuracy look far better than real-world accuracy.

A feature store fixes both problems by defining each feature once, in one place, and reusing it everywhere. On Databricks, feature tables live in Unity Catalog (Feature Engineering in Unity Catalog): any Delta table with a primary key can serve as a feature table, training sets are built with point-in-time lookups, and models logged with feature metadata look up the same features automatically at scoring time.

Analogy — A central kitchen. Instead of every restaurant preparing its own sauce from memory, one central kitchen makes the sauce to one recipe and ships it to every location. The dish tastes the same everywhere because the ingredient is literally the same.

Monitoring — Data Drift, Concept Drift, and Model Decay

Four things to monitor:

  1. DATA QUALITY      Nulls, out-of-range values, schema changes in the
                       inputs. Catches broken pipelines before they reach
                       the model.

  2. DATA DRIFT        The distribution of input features changes.
     (covariate drift) Example: average customer age shifts from 38 to 29
                       after a marketing campaign targets students.

  3. PREDICTION DRIFT  The distribution of predictions changes.
                       Example: the share of customers flagged "high churn
                       risk" jumps from 12% to 31% overnight.

  4. CONCEPT DRIFT     The relationship between inputs and the target
                       changes, even if inputs look the same.
                       Example: a competitor launches a cheaper plan, so
                       price-sensitive customers now churn at a much
                       higher rate than the model learned.

The hard part: true accuracy needs ground truth (did the customer actually
churn?), which often arrives weeks later. Data drift and prediction drift
are early-warning signals you can measure immediately.

Analogy — A weather forecaster who moved cities. A forecaster trained for years in Vancouver moves to Calgary. The same skills, applied to a different climate, produce bad forecasts. The inputs changed (data drift). And even familiar signs mean different things there — a warm wind in winter (a chinook) does not mean the same as it did on the coast (concept drift).

On Databricks, Lakehouse Monitoring can profile Delta tables and inference tables automatically, computing drift metrics into its own metric tables that you can alert on. You can also compute drift yourself, as below.

Measuring Drift with PSI

The Population Stability Index (PSI) compares a feature’s distribution today against its distribution at training time.

Formula (per bin, then summed):
  PSI = Σ (actual% - expected%) × ln(actual% / expected%)

Worked example -- customer age split into 5 bins:

  Bin      Training (expected)   This week (actual)   Contribution
  ───      ───────────────────   ──────────────────   ────────────
  18-25         10%                    5%               0.0347
  26-35         20%                   15%               0.0144
  36-45         40%                   35%               0.0067
  46-55         20%                   25%               0.0112
  56+           10%                   20%               0.0693
                                               PSI =    0.136

Rule-of-thumb thresholds (widely used in credit risk):
  PSI < 0.10         No meaningful change
  0.10 – 0.25        Moderate shift -- investigate
  PSI > 0.25         Significant shift -- likely retrain

  0.136 → moderate drift: older customers are growing as a share.

import numpy as np

def psi(expected, actual, bins=10):
    """Population Stability Index between training and current values."""
    edges = np.quantile(expected, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    expected_pct = np.histogram(expected, edges)[0] / len(expected)
    actual_pct = np.histogram(actual, edges)[0] / len(actual)
    # Avoid log(0) for empty bins
    expected_pct = np.clip(expected_pct, 1e-6, None)
    actual_pct = np.clip(actual_pct, 1e-6, None)
    return float(np.sum((actual_pct - expected_pct) * np.log(actual_pct / expected_pct)))

rng = np.random.default_rng(42)
train_income = rng.normal(60000, 15000, 50000)       # training snapshot
same_income = rng.normal(60000, 15000, 10000)         # this week, unchanged
shifted_income = rng.normal(70000, 18000, 10000)      # this week, shifted

print("PSI unchanged:", round(psi(train_income, same_income), 3))   # 0.001
print("PSI shifted:  ", round(psi(train_income, shifted_income), 3)) # 0.376

Write PSI per feature to a Delta monitoring table after every scoring run, and alert when any important feature crosses 0.25.

Retraining Strategies

A practical retraining job (Lakeflow Jobs or a Fabric pipeline):

  Task 1  Refresh training data         Latest labeled rows into a Delta table
  Task 2  Train challenger              Log run to MLflow, register new version,
                                        set @challenger
  Task 3  Evaluate                      Compare challenger vs champion on the
                                        same recent holdout set
  Task 4  Quality gate                  Promote only if better by a margin AND
                                        passing fairness/sanity checks
  Task 5  Promote or alert              Move @champion, or notify the team

Never auto-promote without a quality gate. "Newer" is not the same as
"better" -- a retrain on a week of corrupted data produces a worse model.

CI/CD for Machine Learning

ML adds two things to normal software CI/CD: data and trained models change independently of code.

Environments (one catalog each in Unity Catalog):

  ml_dev       Experiment freely, small data samples
  ml_staging   Full pipeline on production-like data, automated tests
  ml_prod      Only automated deployments; humans read, jobs write

What to test:
  Code tests       Unit tests for feature functions (pytest)
  Data tests       Schema, null rates, ranges, row counts on training data
  Model tests      Minimum AUC/F1 on a fixed benchmark set; prediction on
                   known examples; latency for real-time endpoints
  Integration      The scoring job runs end to end in staging

How it ships on Databricks:
  Git repo → pull request → CI runs tests → Declarative Automation Bundles
  (DABs) deploy jobs and pipelines to staging → approval → deploy to prod

Our DevOps series covers the deployment mechanics in detail: Databricks Asset Bundles and CI/CD for Databricks with Azure DevOps.

MLOps Maturity Levels

Level 0 -- Manual
  Notebooks, manual training, a model file handed over to be deployed.
  No tracking, no monitoring. Most teams start here.

Level 1 -- Automated training pipeline
  Training runs as a scheduled job on fresh data, every run is tracked,
  models are registered and promoted through a registry, batch scoring is
  automated, basic drift monitoring exists.

Level 2 -- Full CI/CD automation
  Code, data, and model changes all flow through automated testing and
  deployment across dev, staging, and prod. Retraining is triggered by
  monitoring, with automated quality gates and rollback.

Most organizations get the biggest return from moving Level 0 → Level 1.

MLOps on Fabric and Azure Machine Learning

Microsoft Fabric
  ML experiments and ML models are workspace items backed by MLflow.
  Notebooks log runs automatically (autologging is on by default), and
  registered models can score Lakehouse tables in batch with Fabric's
  PREDICT function. Deployment pipelines and Git integration promote
  items across workspaces.

Azure Machine Learning
  Full ML platform with MLflow-compatible tracking and registry, managed
  online endpoints for real-time serving, batch endpoints, and pipelines.
  Common when teams need dedicated ML infrastructure outside a lakehouse.

Databricks
  MLflow fully managed, Unity Catalog registry with aliases and lineage,
  Feature Engineering in Unity Catalog, Model Serving endpoints,
  Lakehouse Monitoring, and Lakeflow Jobs for retraining workflows.

The concepts in this post -- track, register, alias, deploy, monitor,
retrain -- are the same on all three. MLflow is the common language.

Common Mistakes

1. Hard-coding model versions in scoring code. models:/churn_model/7 means every promotion requires a code change and a redeploy. Load by alias (@champion) so promotion and rollback are a single registry update.

2. Computing features differently for training and scoring. Copy-pasting feature logic into a separate SQL view for production is the classic source of training-serving skew. Compute features once, in a shared pipeline or feature table, and use the same code path for both.

3. Overwriting the predictions table every night. Without prediction history you can never measure real accuracy once outcomes arrive, compare model versions, or investigate complaints. Append predictions with a timestamp and the model version.

4. Skipping the model signature. Unity Catalog rejects models without one, and models without a clear input contract break silently when a column is renamed upstream. Always log a signature and an input example.

5. Monitoring only accuracy. Ground truth often arrives weeks late, so accuracy alone detects problems far too slowly. Monitor input data quality, feature drift (PSI), and prediction distribution daily as early warnings.

6. Auto-promoting every retrained model. A retrain on a week of bad data produces a worse model. Every promotion needs a quality gate: the challenger must beat the champion on the same recent holdout, plus sanity checks.

7. Leaking future data into training sets. Joining labels to the current state of a feature table uses information that did not exist at prediction time, inflating offline metrics. Build training sets with point-in-time correct lookups.

8. Treating MLOps as a one-time project. Deployment is the start of a model’s life, not the end. Without owners for monitoring and retraining, every production model eventually decays.

Interview Questions

Q: What is MLOps and why is it needed? A: MLOps is the set of practices for reliably taking machine learning models to production and keeping them working: versioning data, code, and models; automating training and deployment; monitoring performance and drift; and retraining. It is needed because models degrade as real-world data changes, and because production ML systems depend far more on data pipelines and infrastructure than on the model code itself.

Q: What are the main components of MLflow? A: Tracking logs parameters, metrics, artifacts, and models for each run so experiments are reproducible and comparable. The Models component packages models in a standard format with flavors for different libraries and a universal pyfunc interface. The Model Registry versions and governs models — on Databricks it lives in Unity Catalog with aliases, permissions, and lineage. MLflow 3 also adds GenAI evaluation, tracing, and a prompt registry.

Q: What is the difference between model stages and aliases? A: Stages were fixed lifecycle labels (Staging, Production, Archived) in the legacy workspace registry, with only one version per stage. Aliases in Unity Catalog are freely named, movable pointers to versions, such as champion and challenger, and several can exist side by side. Code loads models:/catalog.schema.model@champion, so promotion and rollback are just reassigning an alias. Stages are deprecated in favor of aliases.

Q: What is training-serving skew and how do you prevent it? A: It is a mismatch between how features are computed during training and how they are computed at prediction time, which silently degrades accuracy. Prevent it by defining features once in a shared pipeline or feature store, reusing the same transformation code for training and inference, logging model signatures, and building training sets with point-in-time correct lookups.

Q: Explain data drift versus concept drift. A: Data drift is a change in the distribution of input features — for example, the customer base getting younger. Concept drift is a change in the relationship between inputs and the target — for example, the same customers now churning more because a competitor launched a cheaper plan. Data drift is detectable immediately from inputs; concept drift usually requires ground-truth labels to confirm.

Q: How would you design a batch scoring pipeline? A: A scheduled job runs after the feature tables refresh. It validates the inputs, loads the model by alias from the registry, scores the table in parallel with mlflow.pyfunc.spark_udf, and appends predictions with a timestamp and model version to a Delta table. A follow-up task computes drift metrics such as PSI into a monitoring table and alerts on thresholds. Promotions change the alias, not the job.

Q: How do you decide when to retrain a model? A: Use a combination of a schedule as a safety net, drift triggers based on PSI or prediction distribution changes, and performance triggers when ground truth shows accuracy falling below target. Every retrain produces a challenger that must beat the current champion on the same recent holdout data and pass quality gates before promotion.

Q: What does a data engineer contribute to MLOps? A: Reproducible, versioned training datasets; feature pipelines shared between training and inference; scheduled training and scoring jobs; prediction and monitoring tables; data quality checks on model inputs; environment separation with Unity Catalog; and CI/CD deployment of the whole system. In most teams these pipelines are the majority of the production ML work.

Wrapping Up

MLOps turns a promising notebook into a dependable production system. MLflow tracks every experiment so good models are never lost, signatures define each model’s contract, the Unity Catalog registry versions and governs models, and aliases make promotion and rollback a one-line change. Batch scoring with Spark handles most real workloads, feature stores prevent training-serving skew, drift monitoring catches problems before the business does, and gated retraining keeps models current.

For data engineers, this is familiar territory: pipelines, tables, schedules, environments, and monitoring. The model is just one more artifact flowing through systems you already know how to build well.

Related posts: – Neural Networks and Deep Learning – Model Evaluation – Hyperparameter Tuning – Feature Engineering – Unity Catalog – Databricks Asset Bundles – CI/CD for Databricks

Naveen Vuppula is a Senior Data Engineer and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.