ML Model Deployment

: [University of New South Wales]

- Overview

Machine learning (ML) model deployment is the process of integrating a trained, validated ML model into a production environment so it can accept real-world data and return predictions. It transforms a model from an experimental notebook prototype into an accessible, scalable, and reliable software asset that delivers business value.

The end-to-end process typically follows a structured MLOps (Machine Learning Operations) sequence: Train → Validate → Register → Deploy → Serve → Monitor.

(A) Core Components of Deployment:

A production-ready model deployment relies on three pillars:

Deployment Strategy: Setting up the release pipeline (e.g., handling versioning and testing strategies like A/B tests or canary rollouts).
Model Serving: Providing the runtime infrastructure and network endpoints so external clients can request predictions.
Monitoring & Observability: Tracking inference latency, resource usage, and data/concept drift over time to know when a model needs to be retrained.

(B) Common Strategies for Serving Predictions:

How you deploy your model depends entirely on your data velocity, latency requirements, and budget constraints.

1. Online (Real-time) Inference: The model processes data instantly via an interactive network request and returns immediate answers.

Typical Use Case: Fraud detection, recommendation engines, chatbots.F

Tools Used: FastAPI, Triton, TorchServe

2. Batch Inference: Predictions are run on a scheduled interval (e.g., hourly, nightly) over large collections of data.

Typical Use Case: Generating weekly marketing emails, bulk bank risk assessments.

Tools Used: Apache Spark, Databricks

3. Edge Deployment: The model is compressed and deployed directly onto a client device or web browser.

Typical Use Case: Mobile photo filters, autonomous vehicle object detection.

Tools Used: TensorFlow Lite, ONNX Runtime.

For standard cloud-based online inference, engineering teams typically follow these technical steps:

[Trained Model] ──> [Wrap in API] ──> [Containerize] ──> [Orchestrate/Scale]

Step 1: Save & Register the Model Artifact: Export the trained weights (e.g., .pkl, .onnx, .pt) and log them in a central model registry (like MLflow).
Step 2: Wrap in a Web Framework: Build an API endpoint around the model using lightweight frameworks like FastAPI or Flask. This allows applications to send HTTP POST requests with raw data and receive JSON predictions.
Step 3: Containerization: Package the code, system configurations, and exact library dependencies into a Docker container. This guarantees the model executes identically in testing, staging, and production environments.
Step 4: Scale with Orchestration: Run the container on cloud computing infrastructure. For large-scale traffic, companies use Kubernetes (or specialized tools like KServe) to manage automated horizontal scaling, load balancing, and fault tolerance.

(D) Production Challenges to Keep in Mind:

Moving to production introduces software engineering complications that don't exist in a static research environment:

Model Drift: Over time, real-world data shifts (e.g., changing consumer behavior), causing a model's accuracy to naturally degrade. Continuous data monitoring is required to alert engineers when it is time to retrain.
Compute Constraints: Heavy models (especially deep learning and LLMs) demand significant processing power. Teams use model compression, quantization, and specialized hardware (GPUs/TPUs) to optimize throughput and keep latency down.

- Process to Deploy ML Models

The Machine Learning (ML) deployment process Process to Deploy ML Models bridges the gap between experimentation and real-world application through four foundational steps:

(A) Step 1: Develop and Create a Model in a Training Environment:

Build your model in an offline training environment using training data.
ML teams often create multiple models, but only a few make it to deployment.

(B) Step 2: Optimize and Test Code:

Ensure that your code is of high quality and can be deployed.
Clean and optimize the code as necessary and test it thoroughly to ensure it functions correctly in a live environment.

Containerize your model before deployment.
Containers are predictable, repeatable, and easy to coordinate, making them ideal for deployment.
They simplify deployment, scaling, modification, and updating of ML models.

(D) Step 4: Plan for Continuous Monitoring and Maintenance:

After your model is running, keep checking if it’s working well.
Make sure it still gives good answers and works fast.
If the data changes or it starts making mistakes, fix it.
Update the model often with new information to keep it useful.

- Deployment Strategies

When introducing a new model into production, teams rely on specific testing strategies to mitigate risk and compare performance:

1. Shadow Deployment:

It involves running the new model alongside the existing one without affecting production traffic.
This allows for a comparison of their performances in a real-world setting.
It helps to ensure that the new model meets the required performance metrics before fully deploying it.

2. Canary Deployment:

This means slowly giving the new model to a small group of users while most people keep using the old model.
This way, you can watch how the new model works and find any problems before making it available to everyone.

3. A/B Testing:

It shows different versions of the model to different groups of users and compares how well each one works.
This helps you decide which version is better before using it for all users.

- Best Practices for ML Model Deployment

Always test, version, and secure your machine learning (ML) models automatically before releasing them into production.

1. Automated Testing

Set up CI/CD Pipelines to run tests for code functionality and model quality automatically.
Check data schemas, feature distributions, and edge-case robustness before deployment.
Use Shadow Deployment or canary rollouts to test performance safely on live traffic.

2. Version Control:

Track code changes, datasets, and hyperparameters together to keep training and inference consistent.
Use tools like Git Large File Storage (Git-LFS) or MLflow Model Registry to manage large model binaries.
Containerize applications using Docker to make environments easy to reproduce.

3. Security Measures:

Limit access to models and training data using strict role-based permissions.
Protect prediction endpoints from malicious inputs, data poisoning, and adversarial attacks.
Monitor live systems continuously for data drift or unexpected performance drops.

[More to come ...]

Document Actions

Send this

Sections