Vertex AI Model Deployment: Endpoints, Batch Prediction, Pipelines, and MLOps Best Practices

Note on names: Vertex AI is a machine learning and generative AI platform from Google Cloud. Vertex Cyber Tech Solutions is an independent software, AI, cloud, and cybersecurity services company. We are not affiliated with or endorsed by Google. We build on Vertex AI alongside other platforms.

Vertex AI model deployment is the process of taking a trained model, registering it in Vertex AI Model Registry, and serving it either through an online endpoint for low-latency predictions or through batch prediction jobs for large offline datasets, with autoscaling, versioning, and monitoring managed by Google Cloud. This guide focuses on custom and open models. For calling Gemini, see our Vertex AI development guide.

Online endpoints vs. batch prediction

Step-by-step: deploy a model to an endpoint

Using the Vertex AI SDK for Python (pip install google-cloud-aiplatform):

from google.cloud import aiplatform

aiplatform.init(project="your-project-id", location="us-central1")

model = aiplatform.Model.upload(
    display_name="churn-model",
    artifact_uri="gs://your-bucket/models/churn/v3/",
    # choose a current prebuilt serving container for your framework
    serving_container_image_uri="PREBUILT_SKLEARN_CONTAINER_URI",
)

endpoint = model.deploy(
    machine_type="n1-standard-4",
    min_replica_count=1,
    max_replica_count=3,
)

print(endpoint.predict(instances=[[0.4, 12, 3, 1]]))

Google provides prebuilt serving containers for common frameworks such as scikit-learn, XGBoost, TensorFlow, and PyTorch. Use a custom container when you need your own preprocessing or runtime.

Roll out new versions safely

• Traffic splitting: deploy the new version to the same endpoint and send it a small share of traffic first, then increase it as metrics hold.
• Shadow testing: run the new model in parallel and compare outputs before switching traffic.
• Rollback: keep the previous version deployed until the new one is proven.

Automate with Vertex AI Pipelines

Vertex AI Pipelines runs ML workflows defined with the Kubeflow Pipelines SDK or TFX. A typical pipeline:

1. Extract and validate data, for example from BigQuery.
2. Train and evaluate the model.
3. Compare against the current production model.
4. Register the new version only if it meets thresholds.
5. Deploy, or trigger an approval step for a human reviewer.

Combine this with CI/CD practices from our DevOps guide.

Monitor models in production

• Training-serving skew: production inputs differ from training data.
• Prediction drift: input or output distributions change over time.
• Operational metrics: latency, error rates, and utilization in Cloud Monitoring.
• Business metrics: whether predictions still lead to good outcomes, such as approved transactions that later prove fraudulent.

Vertex AI Model Monitoring can alert on skew and drift so you retrain before accuracy falls.

Cost and security tips

• Right-size machine types and set sensible minimum replicas. Undeploy unused endpoints.
• Use private endpoints and VPC Service Controls for sensitive workloads.
• Restrict who can deploy models with IAM, and log all changes.

Our cloud cost optimization guide covers broader FinOps practices.

Preparing a model for deployment

Before a model reaches an endpoint, it should pass through the same checks every time. A repeatable pre-deployment process includes:

1. Reproducible training: the training code, data snapshot, hyperparameters, and environment are recorded so the model can be rebuilt exactly.
2. Evaluation against a baseline: the candidate model is compared with the current production model on the same held-out data, using the metrics that matter for the business, such as precision at a given recall or mean absolute error in a forecast.
3. Slice analysis: results are broken down by important segments, such as region, customer type, or product line, so a model that is better on average but worse for a key group is caught.
4. Registration: the model artifact, metrics, and metadata are stored in the Vertex AI Model Registry with a version number and labels.
5. Approval: a named owner signs off, either manually or through automated gates that check metrics against thresholds.

Choosing machine types and scaling settings

Endpoint cost is driven mainly by the machine type and how many replicas run. Start with the smallest machine that meets your latency target under realistic load. Many tabular and classical ML models run well on CPU machines; large deep learning models may need GPUs. Set a minimum replica count that covers baseline traffic and a maximum that protects your budget during spikes, and load-test the endpoint to confirm the autoscaling behavior. For workloads with very uneven traffic, consider whether batch prediction or a serverless pattern on Cloud Run would be cheaper than keeping an endpoint always on.

Custom serving containers

Prebuilt containers cover common frameworks such as TensorFlow, PyTorch, scikit-learn, and XGBoost. Use a custom container when you need custom preprocessing, unusual dependencies, or a model server of your choice. The container must expose an HTTP server with health and prediction routes that match the paths you configure. Keep preprocessing logic in the container so training and serving apply exactly the same transformations, which prevents a common class of errors known as training-serving skew. Build containers in CI, scan them for vulnerabilities, and store them in Artifact Registry.

Detecting drift and skew

Models degrade when the data they see in production differs from the data they were trained on. Two types of change matter:

• Training-serving skew: production inputs differ from training data from the start, often because of a preprocessing mismatch or a different data source.
• Prediction drift: production inputs change gradually over time, for example because customer behavior, prices, or seasons change.

Vertex AI Model Monitoring can compare the distribution of incoming features with a training baseline and alert you when differences exceed thresholds. Pair this with business monitoring, such as conversion rates or error rates on decisions the model influences, because drift in inputs does not always mean worse outcomes, and outcomes can worsen without obvious drift.

Retraining strategies

There are three common approaches. Scheduled retraining runs on a fixed cadence, such as monthly, and suits models where data changes steadily. Triggered retraining runs when monitoring detects drift or performance drops below a threshold. Continuous training retrains frequently on new data with automated evaluation and promotion. Whatever you choose, every retrained model should go through the same evaluation, registration, and staged rollout as the first version, so automation never bypasses quality checks.

A CI/CD pipeline for machine learning

MLOps applies software delivery practices to models. A typical pipeline on Google Cloud looks like this:

Define the pipeline in code and keep infrastructure in Terraform so environments are consistent between development, staging, and production.

Example: deploying a demand forecasting model

A retailer forecasts weekly demand for several thousand products. Because forecasts are needed once a week rather than in real time, the team uses batch prediction instead of an always-on endpoint, writing results to BigQuery where the planning team's dashboards read them. A Vertex AI pipeline retrains the model every month on the latest sales data, compares its error against the current model on recent weeks, and only promotes the new version if it is better overall and not worse for the top product categories. Monitoring flags unusual input distributions, such as a sudden spike in promotions data, so analysts can check whether forecasts are still reliable. This design costs far less than a real-time endpoint and keeps the planning team in control.

Rollback and incident response

Things will occasionally go wrong: a new model version underperforms, an upstream data feed breaks, or latency spikes. Prepare by keeping the previous model version deployed or ready to redeploy, documenting how to shift traffic back in one step, and setting alerts on both technical and business metrics. After an incident, record what happened, how it was detected, and what change will prevent it, and add a test or monitor for that failure. For broader DevOps practices that apply equally to ML systems, see our DevOps and CI/CD guide.

Deploying generative models

Many teams now deploy open models, such as Llama or Gemma variants from Model Garden, alongside or instead of hosted Gemini models. The deployment principles are the same, but GPU capacity, quantization, and batching settings have a large effect on cost and latency. Evaluate quality on your own tasks, compare cost per request against hosted models, and consider whether a hosted model with grounding meets your needs before taking on the operational overhead. Our guide to RAG versus fine-tuning explains when a custom model is worth it, and our Vertex AI platform guide covers the wider ecosystem.

Feature management and consistency

Many prediction models depend on features computed from raw data, such as a customer's average order value over the last 90 days or the number of support tickets in the last month. If these features are calculated one way during training and another way at serving time, predictions become unreliable. A feature store, such as Vertex AI Feature Store backed by BigQuery, gives training and serving a shared definition and source for each feature. For simpler projects, a well-tested shared library that computes features in both pipelines can achieve the same consistency. Either way, document each feature's definition, owner, and freshness requirement, because stale features are a frequent cause of silent model degradation.

Explainability and responsible deployment

When model decisions affect people, such as credit limits, prioritization of support tickets, or fraud reviews, stakeholders will ask why the model made a particular prediction. Vertex AI offers feature attribution methods that show which inputs most influenced each prediction. Use these explanations to check that the model relies on sensible signals, to support human reviewers, and to answer audit questions. Combine them with fairness checks on important segments and a clear process for people to challenge automated decisions. Our enterprise AI governance guide covers the policies that should surround deployed models.

MLOps maturity: where are you today?

• Level 0, manual: models are trained in notebooks and deployed by hand. Fine for an initial proof of value, risky for anything business-critical.
• Level 1, automated training: a pipeline trains, evaluates, and registers models on a schedule or trigger, with manual approval for deployment.
• Level 2, automated delivery: code changes, retraining, evaluation, staged rollout, and monitoring all run through CI/CD, with automatic rollback on failure.

Most organizations should aim for level 1 for every production model and level 2 for models that change frequently or drive significant revenue. Moving up a level is usually worth more than adopting another new tool.

Deployment checklist

• Model, data snapshot, code, and metrics are recorded and versioned.
• The candidate beats the production baseline overall and on key segments.
• Serving container is built in CI, scanned, and matches training preprocessing.
• Machine type and autoscaling limits are load-tested.
• Rollout starts with a small traffic share and a documented rollback step.
• Monitoring covers latency, errors, input drift, and business outcomes.
• Endpoint access is restricted with IAM and, where needed, private networking.

Get help operationalizing ML

Vertex Cyber Tech builds production machine learning solutions and MLOps pipelines on Google Cloud, AWS SageMaker, and Azure. Talk to our engineers.

Sources and further reading

• Google Cloud: Overview of getting predictions on Vertex AI
• Google Cloud: Prebuilt containers for prediction
• Google Cloud: Introduction to Vertex AI Pipelines
• Google Cloud: Vertex AI Model Monitoring