Productionize Machine Learning Models with Serverless Container Services | by Edwin Tan | Jan, 2023

By Jessie Hobb On Jan 10, 2023

How to create serverless containerized inference endpoint for your machine learning models with Azure Container App

Serverless container architecture is an approach to building and running containerized applications and services without having to manage the underlying infrastructure. In this architecture, containers are used to package and deploy applications, and these containers are run in a fully managed environment provided by a cloud provider.

The cloud provider is responsible for the infrastructure needed to run the containers, such as the hardware and operating system. The developer does not need to worry about setting up or maintaining this infrastructure, and can instead focus on writing code and building applications. The containers are typically run in a cluster, and the cloud provider automatically scales the number of containers up or down based on demand. This allows the application to handle fluctuations in traffic without the need for manual intervention. Serverless architectures can be more cost-effective than traditional architectures, since users only pay for the resources that was actually use, rather than paying for a fixed amount of computing power that may or may not be fully utilized.

Some examples of serverless container services include Azure Functions, Azure Container Apps , AWS Lambda, and Google Cloud Functions. In this article we will demonstrate how to leverage on Azure Container Apps, a fully managed serverless container service for building and deploying apps at scale, for productionizing machine learning models. Common uses of Azure Container Apps include deploying API endpoints, hosting background processing applications, handling event-driven processing and running microservices [2].

These are the steps that we will take in order to train and deploy a scikit-learn model with Azure Container Apps.

Train model locally
Create inference API with FastAPI
Dockerize the application
Deploy to Azure Container Apps

This is the setup used for the following example.

Development Environment

Visual Studio Code
Azure CLI
Python 3.8
Docker
Python Packages: Refer to Section 3 for requirements.txt

Project Structure

The project folder structure as follows:

FastAPI-Azure-Container-Apps
├─ fastapp
│  └─ app.py
├─ model
│  ├─ heart-disease.joblib
│  └─ train.py
├─ data
│  └─ heart-disease.csv
├─ Dockerfile
├─ requirements.txt
└─ .dockerignore

Dataset

The UCI Heart Disease dataset [3] is a public dataset that contains data about patients who have been diagnosed with heart disease. It includes various patient characteristics, such as age, gender, blood pressure, and cholesterol levels. 1 and 0 values in the condition column represents the presence or absence of heart disease respectively.

File: train.py

For the purpose of illustration, we will train a Gradient Boosting Classifier using only 5 features.

import pathlib
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from joblib import dumpprint ('Load Data')
df = pd.read_csv(pathlib.Path('data/heart-disease.csv'))
y = df.pop('condition')
X = df.loc[:, ['age', 'sex', 'cp', 'trestbps', 'chol']].to_numpy()
print ('Train-Test Split')
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.2)
print ('Train Model')
gbc = GradientBoostingClassifier()
gbc.fit(X_train, y_train)
print ('Save Model')
dump(gbc, pathlib.Path('model/heart-disease.joblib'))

Run the training using CLI:

python model/train.py

File: app.py

from fastapi import FastAPI
from pydantic import BaseModel
import numpy as np
from joblib import load
import pathlibapp = FastAPI(title = 'Heart Disesase Prediction', version = '0.1.0')
model = load(pathlib.Path('model/heart-disease.joblib'))
class ModelData(BaseModel):
age:int=30
sex:int=0
cp:int=2
trestbps:int=115
chol:int=0
class ResponseData(BaseModel):
prediction_result:float=0.1
@app.post('/predict', response_model = ResponseData)
def predict(data:ModelData):
input_data = np.array([v for k,v in data.dict().items()]).reshape(1,-1)
prediction_result = model.predict_proba(input_data)[:,-1]
return {'prediction_result':prediction_result}

In the app.py file we

Define app , an instance of FastAPI.
Load the trained model.
Define the input data format (ModelData) which the API will accept
Define the API response format (ResponseData)
Define the /predict route which will trigger the predict function when a post request is made to this route.
The predict function receives the input data from the post request, predict and returns the probability that a patient has heart disease.

At this point we can test the FastAPI application locally. The --reload flag helps with speeding up the development process. FastAPI auto reloads every time a change in the code is detected. This means that the developer does not need to manually restart FastAPI to test the code changes.

# CLI
uvicorn fastapp.app:app --reload

You will see the following message on the terminal:

INFO:     Uvicorn running on <http://127.0.0.1:8000> (Press CTRL+C to quit)

The given URL will bring us to the Swagger UI where we can test out the API.

Create a requirements.txt file

The requirements.txt file contains all the python packages required.

fastapi>=0.68.0,<0.69.0
pydantic>=1.8.0,<2.0.0
uvicorn>=0.15.0,<0.16.0
numpy == 1.19.5
scikit-learn==0.23.2
joblib==1.1.0
nest_asyncio == 1.5.5

Create a .dockerignore file

The purpose of the .dockerignore file is to avoid copying over files that are not required for inference such as the training script and data.

data/
model/train.py

Create a Dockerfile file

FROM python:3.8
WORKDIR /app
COPY . /app
RUN pip install -r requirements.txt
CMD ["uvicorn", "fastapp.app:app", "--host", "0.0.0.0", "--port", "80"]

Here’s a brief description of the Dockerfile:

Usepython:3.8 as the base image
Creat a working directory named /app
Copy all the files in our project folder over to the working directory with the exception of those files or sub-directories that were listed in the .dockerignore file.
Install the python packages listed in requirements.txt
CMD runs the FastAPI application on port 80 when the docker container is launched. Unlike in the local test, we do not include — reload flag when we run uvicorn here. The reload flag is useful for speeding up the development process, however it is not required in production.

Build docker image

docker build . -t heart-disease-app

Launch docker container

We map port 80 in the container, which FastAPI is running on, to port 8080 on the Docker host.

docker run -p 8080:80 -it heart-disease-app

Test the app

At this point we can test the application through the Swagger UI again by going to the URL: http://127.0.0.1:8080/docs

In this section, we will push the docker image to Azure Container Registry and subsequently deploy the docker container in Azure Container Apps.

To deploy the containerized application to Azure Container Apps we require the following prerequisites:

Azure Resource Group
Azure Container Registry
Azure Container App Environment

The following commands are executed in the command line.

Create resource group

A resource group is a logical grouping of Azure services used to support the application. We created a heartdisease_rg resource group in eastus location. All the subsequent resources will be assigned toheartdisease_rg.

az group create --location eastus --name heartdisease_rg

Create Azure Container Registry

Azure Container Registry (ACR) is a collection of repositories used to store and manage container images. We create a container registry named heartdisease under the heartdisease_rg resource group and chose the Basic SKU pricing plan.

az acr create --name heartdisease --resource-group heartdisease_rg --sku Basic

Once the container registry is provisioned we can check the ACR login server using

az acr list --resource-group heartdisease_rg

The above command returns a long string which contains the login server. Take note of the login server details as we will be using it in the next step.

...
"location": "eastus",
"loginServer": "heartdisease.azurecr.io",
"name": "heartdisease"
...

Tag docker image

To push the local docker image to ACR, the docker image heart-disease-app is tagged the following format {login server}/{docker image name}/{version}.

docker tag heart-disease-app heartdisease.azurecr.io/heart-disease-app:v0.1.0

Login to ACR

Ensure that you are logged in before pushing image to ACR.

az acr login -n heartdisease

Push docker image to ACR

Docker push is a command that uploads a local docker image to a container registry. This will make the docker image available to Azure Container Apps.

docker push heartdisease.azurecr.io/heart-disease-app:v0.1.0

The docker image will be shown in ACR’s UI upon successful docker push.

Create Azure Container App Environment

Before creating Azure Container Apps, we define the heartdiseaseenv environment which the Container App will run in.

az containerapp env create --name heartdiseaseenv --resource-group heartdisease_rg --location eastus

Create Azure Container Apps

In this step, we create the heart-disease-container-app Azure Container Apps using the heartdiseaseenv environment which we created in the step before. We also defined the docker image that should be used: heartdisease.azurecr.io/heart-disease-app:v0.1.0 and the login server of the container registry: heartdisease.azurecr.io. The ingress is set to external as this API is meant to be exposed to the public web.

az containerapp create --name heart-disease-container-app --resource-group heartdisease_rg --environment heartdiseaseenv --image heartdisease.azurecr.io/heart-disease-app:v0.1.0 --target-port 80 --ingress external --query properties.configuration.ingress.fqdn --registry-identity system --registry-server heartdisease.azurecr.io

If az containerapp create command is successful, it will return a URL for accessing your app.

Container app created. Access your app at <https://heart-disease-container-app.nicehill-f0509673.eastus.azurecontainerapps.io/>

Test the app

We can test the app using either the Swagger UI, curl or python request. To access the Swagger UI, simply append docs at the end of the given URL: https://heart-disease-container-app.nicehill-f0509673.eastus.azurecontainerapps.io/docs.

To use CURL, the command as follows:

curl -X 'POST' \\
'<https://heart-disease-container-app.nicehill-f0509673.eastus.azurecontainerapps.io/predict>' \\
-H 'accept: application/json' \\
-H 'Content-Type: application/json' \\
-d '{
"age": 64,
"sex": 1,
"cp": 3,
"trestbps": 120,
"chol": 267
}'

We can also send a post request to the prediction endpoint https://heart-disease-container-app.nicehill-f0509673.eastus.azurecontainerapps.io/predict using python’s request library in the following manner:

import requests
response = requests.post(url = '<https://heart-disease-container-app.nicehill-f0509673.eastus.azurecontainerapps.io/predict>', 
json = {"age": 64,
"sex": 1,
"cp": 3,
"trestbps": 120,
"chol": 267
)
print (response.json())
# {'prediction_result': 0.8298846604381431}

In this article, we discussed the advantage of using serverless containerized machine learning inference endpoint and walked through an example of how to create an API endpoint with FastAPI, containerized it with Docker and deployed the containerized application with Azure Container Apps.

[1] Serverless computing and applications | Microsoft Azure

[2] Azure Container Apps overview | Microsoft Learn

[3] Heart Disease Dataset from UCI Machine Learning Repository. Licensed under CC BY 4.0.

How to create serverless containerized inference endpoint for your machine learning models with Azure Container App

These are the steps that we will take in order to train and deploy a scikit-learn model with Azure Container Apps.

Train model locally
Create inference API with FastAPI
Dockerize the application
Deploy to Azure Container Apps

This is the setup used for the following example.

Development Environment

Visual Studio Code
Azure CLI
Python 3.8
Docker
Python Packages: Refer to Section 3 for requirements.txt

Project Structure

The project folder structure as follows:

FastAPI-Azure-Container-Apps
├─ fastapp
│  └─ app.py
├─ model
│  ├─ heart-disease.joblib
│  └─ train.py
├─ data
│  └─ heart-disease.csv
├─ Dockerfile
├─ requirements.txt
└─ .dockerignore

Dataset

File: train.py

For the purpose of illustration, we will train a Gradient Boosting Classifier using only 5 features.

import pathlib
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import GradientBoostingClassifier
from joblib import dumpprint ('Load Data')
df = pd.read_csv(pathlib.Path('data/heart-disease.csv'))
y = df.pop('condition')
X = df.loc[:, ['age', 'sex', 'cp', 'trestbps', 'chol']].to_numpy()
print ('Train-Test Split')
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.2)
print ('Train Model')
gbc = GradientBoostingClassifier()
gbc.fit(X_train, y_train)
print ('Save Model')
dump(gbc, pathlib.Path('model/heart-disease.joblib'))

Run the training using CLI:

python model/train.py

File: app.py

from fastapi import FastAPI
from pydantic import BaseModel
import numpy as np
from joblib import load
import pathlibapp = FastAPI(title = 'Heart Disesase Prediction', version = '0.1.0')
model = load(pathlib.Path('model/heart-disease.joblib'))
class ModelData(BaseModel):
age:int=30
sex:int=0
cp:int=2
trestbps:int=115
chol:int=0
class ResponseData(BaseModel):
prediction_result:float=0.1
@app.post('/predict', response_model = ResponseData)
def predict(data:ModelData):
input_data = np.array([v for k,v in data.dict().items()]).reshape(1,-1)
prediction_result = model.predict_proba(input_data)[:,-1]
return {'prediction_result':prediction_result}

In the app.py file we

Define app , an instance of FastAPI.
Load the trained model.
Define the input data format (ModelData) which the API will accept
Define the API response format (ResponseData)
Define the /predict route which will trigger the predict function when a post request is made to this route.
The predict function receives the input data from the post request, predict and returns the probability that a patient has heart disease.

# CLI
uvicorn fastapp.app:app --reload

You will see the following message on the terminal:

INFO:     Uvicorn running on <http://127.0.0.1:8000> (Press CTRL+C to quit)

The given URL will bring us to the Swagger UI where we can test out the API.

Create a requirements.txt file

The requirements.txt file contains all the python packages required.

fastapi>=0.68.0,<0.69.0
pydantic>=1.8.0,<2.0.0
uvicorn>=0.15.0,<0.16.0
numpy == 1.19.5
scikit-learn==0.23.2
joblib==1.1.0
nest_asyncio == 1.5.5

Create a .dockerignore file

The purpose of the .dockerignore file is to avoid copying over files that are not required for inference such as the training script and data.

data/
model/train.py

Create a Dockerfile file

FROM python:3.8
WORKDIR /app
COPY . /app
RUN pip install -r requirements.txt
CMD ["uvicorn", "fastapp.app:app", "--host", "0.0.0.0", "--port", "80"]

Here’s a brief description of the Dockerfile:

Usepython:3.8 as the base image
Creat a working directory named /app
Copy all the files in our project folder over to the working directory with the exception of those files or sub-directories that were listed in the .dockerignore file.
Install the python packages listed in requirements.txt
CMD runs the FastAPI application on port 80 when the docker container is launched. Unlike in the local test, we do not include — reload flag when we run uvicorn here. The reload flag is useful for speeding up the development process, however it is not required in production.

Build docker image

docker build . -t heart-disease-app

Launch docker container

We map port 80 in the container, which FastAPI is running on, to port 8080 on the Docker host.

docker run -p 8080:80 -it heart-disease-app

Test the app

At this point we can test the application through the Swagger UI again by going to the URL: http://127.0.0.1:8080/docs

In this section, we will push the docker image to Azure Container Registry and subsequently deploy the docker container in Azure Container Apps.

To deploy the containerized application to Azure Container Apps we require the following prerequisites:

Azure Resource Group
Azure Container Registry
Azure Container App Environment

The following commands are executed in the command line.

Create resource group

az group create --location eastus --name heartdisease_rg

Create Azure Container Registry

az acr create --name heartdisease --resource-group heartdisease_rg --sku Basic

Once the container registry is provisioned we can check the ACR login server using

az acr list --resource-group heartdisease_rg

The above command returns a long string which contains the login server. Take note of the login server details as we will be using it in the next step.

...
"location": "eastus",
"loginServer": "heartdisease.azurecr.io",
"name": "heartdisease"
...

Tag docker image

To push the local docker image to ACR, the docker image heart-disease-app is tagged the following format {login server}/{docker image name}/{version}.

docker tag heart-disease-app heartdisease.azurecr.io/heart-disease-app:v0.1.0

Login to ACR

Ensure that you are logged in before pushing image to ACR.

az acr login -n heartdisease

Push docker image to ACR

Docker push is a command that uploads a local docker image to a container registry. This will make the docker image available to Azure Container Apps.

docker push heartdisease.azurecr.io/heart-disease-app:v0.1.0

The docker image will be shown in ACR’s UI upon successful docker push.

Create Azure Container App Environment

Before creating Azure Container Apps, we define the heartdiseaseenv environment which the Container App will run in.

az containerapp env create --name heartdiseaseenv --resource-group heartdisease_rg --location eastus

Create Azure Container Apps

az containerapp create --name heart-disease-container-app --resource-group heartdisease_rg --environment heartdiseaseenv --image heartdisease.azurecr.io/heart-disease-app:v0.1.0 --target-port 80 --ingress external --query properties.configuration.ingress.fqdn --registry-identity system --registry-server heartdisease.azurecr.io

If az containerapp create command is successful, it will return a URL for accessing your app.

Container app created. Access your app at <https://heart-disease-container-app.nicehill-f0509673.eastus.azurecontainerapps.io/>

Test the app

To use CURL, the command as follows:

curl -X 'POST' \\
'<https://heart-disease-container-app.nicehill-f0509673.eastus.azurecontainerapps.io/predict>' \\
-H 'accept: application/json' \\
-H 'Content-Type: application/json' \\
-d '{
"age": 64,
"sex": 1,
"cp": 3,
"trestbps": 120,
"chol": 267
}'

import requests
response = requests.post(url = '<https://heart-disease-container-app.nicehill-f0509673.eastus.azurecontainerapps.io/predict>', 
json = {"age": 64,
"sex": 1,
"cp": 3,
"trestbps": 120,
"chol": 267
)
print (response.json())
# {'prediction_result': 0.8298846604381431}

[1] Serverless computing and applications | Microsoft Azure

[2] Azure Container Apps overview | Microsoft Learn

[3] Heart Disease Dataset from UCI Machine Learning Repository. Licensed under CC BY 4.0.

Read original article here

Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.