Deploy TensorFlow ResNet50 on SageMaker Inference
In the past I’ve written about deploying pre-trained models on Amazon SageMaker. While the artifact I published is replicable it requires understanding of AWS SDKs and the appropriate API calls, along with an understanding of the higher level SageMaker Python SDK.
This can be a lot to digest for new users and to make the experience simpler, I’ve helped build an API that automates a lot of the lower level work a user had to understand to properly deploy a SageMaker endpoint. With this API, the focus is on making the migration of pre-trained models a lot simpler and intuitive for Data Scientists and users new to SageMaker or AWS in general.
What’s supported at the moment? Currently the API is known as the SageMaker Migration Toolkit and it supports pre-trained model deployment of TensorFlow, PyTorch, and Sklearn models. Along with this there’s support for both Real-Time and Serverless endpoints, by default the package deploys to Serverless endpoints if you don’t specify the inference option you would like. Lastly, only single model endpoints are supported for real-time inference at the moment, but for manual examples of pre-trained Multi-Model deployment check out these examples.
In today’s article I’ll walk you through on how you can use this API with the popular pre-trained ResNet50 Model utilizing the TensorFlow framework.
SageMaker Migration Toolkit Setup
Instructions for setup can be found on the GitHub readme for the API. Installation is very similar to other Python packages as you’ll see in the following steps.
- Clone GitHub repo
- python setup.py bdist_wheel
- pip install dist/sagemaker_migration_toolkit-0.0.1-py3-none-any.whl
- sagemaker_migration-configure — module-name sagemaker_migration.configure
The fourth step is where you can add the IAM role for the SageMaker service. You should be able to access this in your IAM console or run the following code in a notebook cell to get your role.
import sagemaker
sagemaker.get_execution_role()
For this example we’re going to be working in a SageMaker notebook with a conda_tf2 kernel so that TensorFlow comes pre-installed.
ResNet50 Model Setup
Before we can get to using the Migration Toolkit, we need to properly download and serialize the ResNet50 model artifacts. SageMaker expects the model data to be in a format compliant with the framework’s serving stack. For TensorFlow Serving the serialized model data follows the SavedModel format. The structure of the model data should look like the following:
model_artifact_version/
variables/
variables.data
variables.index
keras_metadata.pb
saved_model.pb
inference.py #optional, but recommended
Using TensorFlow we can load the ResNet50 model.
import os
import tensorflow as tf
from tensorflow.keras.applications import resnet50
from tensorflow.keras import backend
import numpy as np
from tensorflow.keras.preprocessing import imagetf.keras.backend.set_learning_phase(0)
model = resnet50.ResNet50() #Load model
Now that our model is loaded we can dump these model artifacts in a directory with the structure shown above.
export_dir = "00002" #directory to store model artifacts
model = tf.keras.applications.ResNet50()if not os.path.exists(export_dir):
os.makedirs(export_dir)
print("Directory ", export_dir, " Created ")
else:
print("Directory ", export_dir, " already exists")
# Save to SavedModel
model.save(export_dir, save_format="tf", include_optimizer=False)
Lastly, the SageMaker Migration Toolkit expects an inference.py script to be provided as well. For those unfamiliar an inference.py file allows for users to add their pre and post processing logic to their endpoint. For this model we don’t really need any pre/post processing so we can create a dummy inference script that does not have any functionality.
%%writefile inference.py
import os
import json
We’re now ready to utilize the Migration Toolkit for model deployment.
SageMaker Migration Toolkit Deployment
We first import the necessary module for utilizing the package.
from sagemaker_migration import frameworks as fwk
import os
Within the frameworks module, there’s currently support for TensorFlow, PyTorch, and Sklearn. We create a TensorFlow Model Entity and pass in the appropriate parameters.
# Create a TensorFlow Model Object Entity, you can create a real-time or serverless endpoint
tf_model = fwk.TensorFlowModel(
version = '2.3.0',
model_data = '00002',
inference_option = 'real-time',
inference = 'inference.py',
instance_type = 'ml.m5.xlarge')
Here there’s a few mandatory parameters:
- model_data (str): This is the path to your serialized model data.
- inference (str): This is your inference.py script, provide one even if it is empty.
- version (str): This is the framework version of your model, underneath the hood the toolkit will search for a supported SageMaker container of that version.
As for inference_option the default is set to serverless inference. If you specify the inference option you can also set parameters such as instance type for Real-Time Inference and then Concurrency/Memory for Serverless Inference.
The main difference from utilizing Boto3 is that you no longer have to interact with model, endpoint configuration, images, and other lower level related SageMaker objects. When you deploy this TensorFlow model entity you will see all of these being created automatically.
tf_model.deploy_to_sagemaker()
The model tarball is automatically created for you, as is the SageMaker Model, SageMaker Endpoint Configuration, and finally the SageMaker Endpoint itself.
Conclusion
You can find the code for this example at the link above. If you have any problems utilizing the API or any models you would like to request please add a PR or Issue on the Github. As always I hope the article was a useful guide, especially to those new to model deployment on SageMaker. Stay tuned for more and all feedback is appreciated.
Additional Resources
Load Testing SageMaker Endpoints
SageMaker Bring Your Own Container
Deploy TensorFlow ResNet50 on SageMaker Inference
In the past I’ve written about deploying pre-trained models on Amazon SageMaker. While the artifact I published is replicable it requires understanding of AWS SDKs and the appropriate API calls, along with an understanding of the higher level SageMaker Python SDK.
This can be a lot to digest for new users and to make the experience simpler, I’ve helped build an API that automates a lot of the lower level work a user had to understand to properly deploy a SageMaker endpoint. With this API, the focus is on making the migration of pre-trained models a lot simpler and intuitive for Data Scientists and users new to SageMaker or AWS in general.
What’s supported at the moment? Currently the API is known as the SageMaker Migration Toolkit and it supports pre-trained model deployment of TensorFlow, PyTorch, and Sklearn models. Along with this there’s support for both Real-Time and Serverless endpoints, by default the package deploys to Serverless endpoints if you don’t specify the inference option you would like. Lastly, only single model endpoints are supported for real-time inference at the moment, but for manual examples of pre-trained Multi-Model deployment check out these examples.
In today’s article I’ll walk you through on how you can use this API with the popular pre-trained ResNet50 Model utilizing the TensorFlow framework.
SageMaker Migration Toolkit Setup
Instructions for setup can be found on the GitHub readme for the API. Installation is very similar to other Python packages as you’ll see in the following steps.
- Clone GitHub repo
- python setup.py bdist_wheel
- pip install dist/sagemaker_migration_toolkit-0.0.1-py3-none-any.whl
- sagemaker_migration-configure — module-name sagemaker_migration.configure
The fourth step is where you can add the IAM role for the SageMaker service. You should be able to access this in your IAM console or run the following code in a notebook cell to get your role.
import sagemaker
sagemaker.get_execution_role()
For this example we’re going to be working in a SageMaker notebook with a conda_tf2 kernel so that TensorFlow comes pre-installed.
ResNet50 Model Setup
Before we can get to using the Migration Toolkit, we need to properly download and serialize the ResNet50 model artifacts. SageMaker expects the model data to be in a format compliant with the framework’s serving stack. For TensorFlow Serving the serialized model data follows the SavedModel format. The structure of the model data should look like the following:
model_artifact_version/
variables/
variables.data
variables.index
keras_metadata.pb
saved_model.pb
inference.py #optional, but recommended
Using TensorFlow we can load the ResNet50 model.
import os
import tensorflow as tf
from tensorflow.keras.applications import resnet50
from tensorflow.keras import backend
import numpy as np
from tensorflow.keras.preprocessing import imagetf.keras.backend.set_learning_phase(0)
model = resnet50.ResNet50() #Load model
Now that our model is loaded we can dump these model artifacts in a directory with the structure shown above.
export_dir = "00002" #directory to store model artifacts
model = tf.keras.applications.ResNet50()if not os.path.exists(export_dir):
os.makedirs(export_dir)
print("Directory ", export_dir, " Created ")
else:
print("Directory ", export_dir, " already exists")
# Save to SavedModel
model.save(export_dir, save_format="tf", include_optimizer=False)
Lastly, the SageMaker Migration Toolkit expects an inference.py script to be provided as well. For those unfamiliar an inference.py file allows for users to add their pre and post processing logic to their endpoint. For this model we don’t really need any pre/post processing so we can create a dummy inference script that does not have any functionality.
%%writefile inference.py
import os
import json
We’re now ready to utilize the Migration Toolkit for model deployment.
SageMaker Migration Toolkit Deployment
We first import the necessary module for utilizing the package.
from sagemaker_migration import frameworks as fwk
import os
Within the frameworks module, there’s currently support for TensorFlow, PyTorch, and Sklearn. We create a TensorFlow Model Entity and pass in the appropriate parameters.
# Create a TensorFlow Model Object Entity, you can create a real-time or serverless endpoint
tf_model = fwk.TensorFlowModel(
version = '2.3.0',
model_data = '00002',
inference_option = 'real-time',
inference = 'inference.py',
instance_type = 'ml.m5.xlarge')
Here there’s a few mandatory parameters:
- model_data (str): This is the path to your serialized model data.
- inference (str): This is your inference.py script, provide one even if it is empty.
- version (str): This is the framework version of your model, underneath the hood the toolkit will search for a supported SageMaker container of that version.
As for inference_option the default is set to serverless inference. If you specify the inference option you can also set parameters such as instance type for Real-Time Inference and then Concurrency/Memory for Serverless Inference.
The main difference from utilizing Boto3 is that you no longer have to interact with model, endpoint configuration, images, and other lower level related SageMaker objects. When you deploy this TensorFlow model entity you will see all of these being created automatically.
tf_model.deploy_to_sagemaker()
The model tarball is automatically created for you, as is the SageMaker Model, SageMaker Endpoint Configuration, and finally the SageMaker Endpoint itself.
Conclusion
You can find the code for this example at the link above. If you have any problems utilizing the API or any models you would like to request please add a PR or Issue on the Github. As always I hope the article was a useful guide, especially to those new to model deployment on SageMaker. Stay tuned for more and all feedback is appreciated.
Additional Resources
Load Testing SageMaker Endpoints
SageMaker Bring Your Own Container