In the previous blog How to level up your MLOps skills with GCP and get familiar with essential tools — Part 1. I went through an over view of MLOps and the essential tools to get familiar with before you implement a CI/CD cycle to your ML based solutions, as promised now I will take you through and end-to-end deployment lifecycle overview with GCP, for this I will be referencing the main GCP page where most of the figures and resources are found with details.
So, grab your cup of coffee or tea perhaps 😉 and enjoy reading!
There are key milestones that you need to make sure you meet by building the cycle, these are : managing features via Vertex AI Feature store, managing models using Vertex AI Model Registry, evaluating models, orchestrate ML workflows using pipelines, track and analyse your ML metadata, understand model behaviour, track model quality and finally track experiments. That been said lets dive into a lifecycle:
Vertex AI Feature Store data model:
The feature values are are stored using a time series data model in Vertex AI Feature Store. The reason behind having a time series based model is to help maintaining feature values even if they change over time. Vertex AI Resources are organised hierarchically in the following order: Featurestore -> EntityType -> Feature. These resources have to be created before ingesting data into the feature store. To understand this let’s have an example, given the following sample source data in a BigQuery table,
The following source data is about movies and their features.
Before data ingestion a top-level container should be created a featurestore, Then in this featurestore, you need to create entity types that group and contain related features. Then your features can be created and mapped to features in your source data.
In the above image the movie_id column can map to an entity type movie. The average_rating, title, and genre are features of the movie entity type. The values in each column map to specific instances of an entity type or features, which are called entities and feature values.
The update_time column has the feature values generation time. In the featurestore, this timestamp column is not required as you can specify the timestamp as part of the ingestion request. Furthermore, if you want to preform feature data ingestion, Vertex AI Feature Store can ingest data from tables in BigQuery or files in Cloud Storage, but I am not going to dive into the details of source data requirements, you can go through it with details here.
One important thing that I need to mention about the feature store is the online and offline storage methods of Vertex AI Feature Store. All featurestores have offline storage and optionally, online storage. The benefit of the Online storing method is that Online storage saves the latest timestamp values features to handle online serving requests. When you use the API to preform an ingestion job you can check if the data is written to the online store, otherwise in the case of offline storage there would not be any load on the online serving nodes. The online storage can be so helpful when preforming backfill jobs. Data is stored in Vertex AI Feature Store in offline manners until the data reaches the retention limit or until its deleted, there is no limit to the amount of the data that you can store offlline.
So far we went through the feature management part with the feature store of the process but what about managing the models? that makes me move to the next part.
Vertex AI Model Registry:
To manage ML models the Vertex AI Model Registry is used as a central repository where the lifecycle of ML models can be managed. From the Vertex AI Model Registry, it gives an overview of your models and organise, track, and train new versions. Model versions that need to be deployed can be assigned an endpoint from the registry, or using aliases, deploy models to an endpoint.
To add your models to the registry you have to follow the following steps:
- Import models to the registry.
- Create new models, assign a model version the default alias, ready for production.
- You can also add other aliases/labels to help managing and organising your model versions.
- Deploy your models to an endpoint.
- Run batch prediction, and start your model evaluation pipeline.
- View the model performance metrics from the model details page.
If you want to have a hands on experience with some coding to deploy BiqQuery ML Model on Vertex AI Model Registry and make predictions you can check this notebook. Also, if you want to preform different procedures like copying, deleting or integrating with BigQuery ML you can read more here.
Evaluating and orchestrating models:
Now, let’s move to how to evaluate your model with Vertex AI, you can preform precision and recall to evaluate the performance of your models.
The model evaluation provided by Vertex AI can fit in the typical machine learning workflow, one interesting enhancement to the model evaluation process is discussed in the following article. Follow the steps over here to preform model evaluation You can also have a look at the Continuous Model Evaluation workflow approach in the diagram attached down:
Okay, what about orchestrating the ML workflow on Vertex AI Pipelines, Vertex AI deals with pipelines, so you need to describe your workflow as a pipeline. A pipeline is a portable, scalable ML workflow that is based on a container. ML pipelines are composed of a set of input parameters and a list of steps each step represents a pipeline component.
You remember the tools Kubeflow and Tensorflow Extended that I mentioned in the previous post? you can use Vertex AI Pipelines to run pipelines that were built using these two tools. Learn more about choosing between the Kubeflow Pipelines SDK and TFX.
In order to understand changes and monitor the performance of your ML system, you need to analyse the metadata of pipeline runs and the lineage of ML artifacts. What is an artifact’s lineage? An artifact’s lineage includes all the factors in the model creation as well as artifacts and metadata that are derived from this artifact. That been said, let us have a look at how the metadata is analysed in the following section.
Analysing metadata & understanding the model behaviour:
Building an ML pipeline it is critical to track the parameters, artifacts, and metrics used in the experiment when training and evaluating the model.
Vertex ML Metadata lets you record the metadata and artifacts produced by your ML pipeline also it enables you to query that metadata to analyse, debug, and audit the performance of your ML system. For this Vertex ML uses the open source ML Metadata (MLMD) library that was developed by Google’s TensorFlow Extended team. The figure below gives an overview to where the metadata validation takes part in the whole system.
Even though my blog is not diving into deep details about how to actually code and connect all the parts I came across this good code snippets on how to fetch the metadata that is stored in GCP BigQuery using BigQuery client and perform the metadata checks as follows:
from google.cloud import bigqueryclient = bigquery.Client(project=PROJECT_ID)
# define a lambda function to get the delta days
get_delta_day = lambda td: td.days
# perform general checks
for bq_table in bq_tables:
table = client.get_table(bq_table)
print(
"Got table '{}.{}.{}'.".format(
table.project, table.dataset_id, table.table_id
)
)
print("Table schema: {}".format(table.schema))
print("Table description: {}".format(table.description))
print("Table has {} rows".format(table.num_rows))
assert table.num_rows > 0, "Table has no rows"
assert (
get_delta_day(datetime.now(pytz.utc) - table.modified) < 30
), "Table is not updated in the last 30 days"
# Table specific checks
table1 = client.get_table("table_name")
# add example assert here
assert (
get_delta_day(datetime.now(pytz.utc) - table1.modified) < 3
), "Table is not updated in the last 3 days"
You can have a look at the main documentation about this topic here.
Track model quality & experiments:
The model performance changes according to the data used for prediction the larger the difference between training data and the data used for prediction the more the model’s performance deteriorate, even if the model itself hasn’t changed.
The model’s prediction input data can be monitored for feature skew and drift:
- Training-serving skew is what happens when your feature data distribution used in production is different from the one used to train the model. The skew detection can be preformed if the data used for training is accessible.
- Prediction drift is what happens there’s a significant change in the feature data distribution in production, if the data used for training wasn’t accessible, drift detection could be enabled to monitor the input data for changes over time.
Feature skew and drift detection is supported by Model Monitoring for both categorical and numerical features, Once the skew or drift for a model’s feature exceeds a defined alerting threshold, Model Monitoring sends an email alert. It also offers a view of the distribution of features over time and that helps in making the decision on wether the model should be retrained or not.
Have a look on the skew or drift between the baseline and latest distributions of a numerical feature:
You can provide schemas to Vertex AI Model Monitoring, that could be for online prediction endpoints or for batch predictions.
When creating a Model Monitoring job you can provide your own input schema to guarantee that Model Monitoring correctly parses your model’s inputs. This schema is called analysis instance schema. The schema file specifies the format of the input payload, the names of each feature, and the type of each feature and is written in YAML file in the Open API format. Have a look at the example provided here.
Finally, Vertex AI Experiments is a tool used to analyse and track different model architectures, hyperparameters, and training environments. Vertex AI Experiments can also evaluate the model performance in aggregate, against test datasets, and during the training run. This helps selecting the best model according to the use-case. Here’s a full guid on how to set up your own Vertex AI experiment.
Now we finished and went through all the functionalities that Vertex AI provides to build an end to end ML system. To have a holistic grouping of all of these steps eventually we can say the whole process is grouped into:
Experimentation & Packaging → Model Versioning → CI (Continuous Integration) → CT (Continuous Training) → CD (Continuous Delivery) → Monitoring
In the first step tools like MLFlow, Kubeflow, Tensorflow Extended and Jupyter notebooks for data cleaning and the data science part of the process, for the model versioning Git can be used of course the Model registry plays a big role here, the Continuous Integration is handled by Vertex AI’s components like Model registry, Feature store, ML metadata store, ML pipeline orchestrator and Source Control, Test and build services, Deployment services. In the Continuous Training phase, the machine learning models is automatically orchestrated over Kubernetes cluster via Kubeflow Pipelines.
In the Continuous Delivery stage, the trained model is put into a production-ready environment, the complexity of auto-scaling and server configuration, networking could be handled with Kubeflow native KFServing, which enables server-less inferencing on Kubernetes and provides high performant and easy abstraction interfaces for common machine learning frameworks. Finally, for monitoring I can say it matches the model quality and experiments tracking section discovered previously in this blog, where real-time inference logs can be stored with various details about confidence score for some predictions, IP-address of the clients using services.
Thats it for this post, I hope you enjoyed reading and now have an overview on the whole ML system maintaining/deploying life cycle.
I heavily used the main MLOps documentation with Vertex AI on the main Google Cloud website where you can find more details and the source images used and referenced.
