A Flexible End-to-End Template for Amazon SageMaker Projects | by Lars Kjeldgaard | Sep, 2022

In this article, I will go through how to use the new ‘shapemaker’ template for creating Amazon SageMaker projects with maximum flexibility.

This article targets full-stack data scientists with intermediate knowledge of python, Amazon SageMaker as well as AWS in general, docker, shell scripting and development of web applications.

Why should you care?
Amazon SageMaker manages containers for popular frameworks like TensorFlow, PyTorch and HuggingFace, you can use out-of-the-box for creating model training jobs and endpoints for inference. This allows developers to focus only on providing a training and an inference script (i.e. working in Script Mode).

From my experience with real-life production settings however, it is often the case, this approach does not offer enough flexibility: maybe (1) the framework you are working with is not supported by SageMaker’s existing containers, (2) you need to customize the training job container or (3) you need to customize the endpoint container (or how it is served).

The solution: Bring Your Own Container
To address this SageMaker offers a functionality named Bring Your Own Container (BYOC) providing full developer control. As the title hints, BYOC means you bring SageMaker your own containers for training jobs and inference endpoints, which you can then use through the SageMaker API.

At first sight working with Amazon Sagemaker BYOC can seem like quite a mouthful. Especially if you are used to working in Script Mode.

In an attempt to make BYOC easier to grasp, I have forged a template for Amazon SageMaker projects, that implements BYOC.

Introducing ‘shapemaker’
shapemaker is a complete end-to-end template for Amazon SageMaker AWS projects aiming for maximum flexibility. It builds on the BYOC SageMaker functionality for full developer control.

The template includes:

• a minimalistic template for model code
• a template for a docker image for model training
• an endpoint docker image template for real-time inference
• command-line functions for interacting with the model/endpoint
• command-line functions for delivering and integrating the model w/SageMaker
• continuous integration/delivery workflows.

‘shapemaker’ supports Linux/macOS.

You can create a new project from the ‘shapemaker’ template with cookiecutter:

cookiecutter gh:smaakage85/shapemaker

The video below gives a super quick walkthrough of some of the most important features of the ‘shapemaker’ template and how to use it: it goes through how to create a project from the template and how to use its built-in command-line functions to build training and endpoint images as well as creating training jobs and endpoints. Furthermore I show how to enable shapemaker CI/CD workflows.

NOTE: For those of you new to AWS, make sure you make an account at the following link if you want to follow along. There will be costs incurred through the deployment process, especially if you leave your endpoint up and running.

In the following I will go through some of the nuts and bolts of the ‘shapemaker’ template.

For more detailed instructions on how to use ‘shapemaker’, please refer to the README.

1. Model code

The model code is organized as a python package: modelpkg.

The template ships with a dummy model serving as a placeholder: a model estimating a linear relationship between salary and age.

The model is implemented as its own class with methods for (1) model training, (2) predicting observations, (3) performance evaluation and (4) loading/saving model artifacts.

2. Model training image
The training script train.pydraws upon the model code in modelpkg. train.py runs a parametrized training job producing and saving a model artifact.

modelpkg and the training script above are build into the model training docker image capable of running a model training job.

3. Endpoint image
For the model endpoint image ‘shapemaker’ ships with a Flask web app, that uses modelpkg to load a model artifact and compute predictions as answers to requests from the users of the app.

The app builds into the endpoint docker image. By default ‘shapemaker’ implements an NGINX frontend.

4. Build, train and deploy model with command-line functions
All tasks related to interacting with the model project are implemented as convenient command-line functions in Makefile implying that functions are invoked with make [target], e.g. make build_training_image.

If you want to build, train and deploy a model on-the-fly you can do it by invoking a sequence of make targets, i.e.:

1. make init
2. make build_training_image
3. make push_training_image
4. make create_training_job
5. make build_endpoint_image
6. make push_endpoint_image
7. make create_endpoint

You can delete the endpoint afterwards by invoking make delete_endpoint .

NOTE: make + space + tab + tab lists all available make targets.

5. Configuration files
The configurations for training jobs, endpoints etc. are separated from the code and reside in the configs folder. This implies, that if you for instance want to make a change to how the endpoint is configured, you only need to make a change to configs/endpoint_config.json:

6. CI/CD workflows
shapemaker comes with a number of automation (CI/CD) workflows implemented with Github Actions.

To enable CI/CD workflows, upload your project to Github and connect the Github repository with your AWS account by providing your AWS credentials as Github Secrets. Secrets should have names:

1. AWS_ACCESS_KEY_ID
2. AWS_SECRET_ACCESS_KEY

By default, every commit to main triggers a workflow ./github/workflows/deliver_images.yaml, that runs unit tests and builds and pushes training and endpoint images:

All workflows can be run manually.

Conclusion
BYOC gives incredible flexibility to configure and adjust SageMaker projects but comes at the expense of increased complexity.

In an attempt to make BYOC easier to grasp, ‘shapemaker’ offers a complete end-to-end template for Amazon Sagemaker projects, that implements BYOC.

I hope, you will give ‘shapemaker’ a spin. If you do, I would love your feedback.

Resources
This blog post stands on the shoulders of (and borrows from) the following blog posts by Maria Vexlard and Ram Vegiraju — big shout-outs to them:

In this article, I will go through how to use the new ‘shapemaker’ template for creating Amazon SageMaker projects with maximum flexibility.

This article targets full-stack data scientists with intermediate knowledge of python, Amazon SageMaker as well as AWS in general, docker, shell scripting and development of web applications.

Why should you care?
Amazon SageMaker manages containers for popular frameworks like TensorFlow, PyTorch and HuggingFace, you can use out-of-the-box for creating model training jobs and endpoints for inference. This allows developers to focus only on providing a training and an inference script (i.e. working in Script Mode).

From my experience with real-life production settings however, it is often the case, this approach does not offer enough flexibility: maybe (1) the framework you are working with is not supported by SageMaker’s existing containers, (2) you need to customize the training job container or (3) you need to customize the endpoint container (or how it is served).

The solution: Bring Your Own Container
To address this SageMaker offers a functionality named Bring Your Own Container (BYOC) providing full developer control. As the title hints, BYOC means you bring SageMaker your own containers for training jobs and inference endpoints, which you can then use through the SageMaker API.

At first sight working with Amazon Sagemaker BYOC can seem like quite a mouthful. Especially if you are used to working in Script Mode.

In an attempt to make BYOC easier to grasp, I have forged a template for Amazon SageMaker projects, that implements BYOC.

Introducing ‘shapemaker’
shapemaker is a complete end-to-end template for Amazon SageMaker AWS projects aiming for maximum flexibility. It builds on the BYOC SageMaker functionality for full developer control.

The template includes:

• a minimalistic template for model code
• a template for a docker image for model training
• an endpoint docker image template for real-time inference
• command-line functions for interacting with the model/endpoint
• command-line functions for delivering and integrating the model w/SageMaker
• continuous integration/delivery workflows.

‘shapemaker’ supports Linux/macOS.

You can create a new project from the ‘shapemaker’ template with cookiecutter:

cookiecutter gh:smaakage85/shapemaker

The video below gives a super quick walkthrough of some of the most important features of the ‘shapemaker’ template and how to use it: it goes through how to create a project from the template and how to use its built-in command-line functions to build training and endpoint images as well as creating training jobs and endpoints. Furthermore I show how to enable shapemaker CI/CD workflows.

NOTE: For those of you new to AWS, make sure you make an account at the following link if you want to follow along. There will be costs incurred through the deployment process, especially if you leave your endpoint up and running.

In the following I will go through some of the nuts and bolts of the ‘shapemaker’ template.

For more detailed instructions on how to use ‘shapemaker’, please refer to the README.

1. Model code

The model code is organized as a python package: modelpkg.

The template ships with a dummy model serving as a placeholder: a model estimating a linear relationship between salary and age.

The model is implemented as its own class with methods for (1) model training, (2) predicting observations, (3) performance evaluation and (4) loading/saving model artifacts.

2. Model training image
The training script train.pydraws upon the model code in modelpkg. train.py runs a parametrized training job producing and saving a model artifact.

modelpkg and the training script above are build into the model training docker image capable of running a model training job.

3. Endpoint image
For the model endpoint image ‘shapemaker’ ships with a Flask web app, that uses modelpkg to load a model artifact and compute predictions as answers to requests from the users of the app.

The app builds into the endpoint docker image. By default ‘shapemaker’ implements an NGINX frontend.

4. Build, train and deploy model with command-line functions
All tasks related to interacting with the model project are implemented as convenient command-line functions in Makefile implying that functions are invoked with make [target], e.g. make build_training_image.

If you want to build, train and deploy a model on-the-fly you can do it by invoking a sequence of make targets, i.e.:

1. make init
2. make build_training_image
3. make push_training_image
4. make create_training_job
5. make build_endpoint_image
6. make push_endpoint_image
7. make create_endpoint

You can delete the endpoint afterwards by invoking make delete_endpoint .

NOTE: make + space + tab + tab lists all available make targets.

5. Configuration files
The configurations for training jobs, endpoints etc. are separated from the code and reside in the configs folder. This implies, that if you for instance want to make a change to how the endpoint is configured, you only need to make a change to configs/endpoint_config.json:

6. CI/CD workflows
shapemaker comes with a number of automation (CI/CD) workflows implemented with Github Actions.

To enable CI/CD workflows, upload your project to Github and connect the Github repository with your AWS account by providing your AWS credentials as Github Secrets. Secrets should have names:

1. AWS_ACCESS_KEY_ID
2. AWS_SECRET_ACCESS_KEY

By default, every commit to main triggers a workflow ./github/workflows/deliver_images.yaml, that runs unit tests and builds and pushes training and endpoint images:

All workflows can be run manually.

Conclusion
BYOC gives incredible flexibility to configure and adjust SageMaker projects but comes at the expense of increased complexity.

In an attempt to make BYOC easier to grasp, ‘shapemaker’ offers a complete end-to-end template for Amazon Sagemaker projects, that implements BYOC.

I hope, you will give ‘shapemaker’ a spin. If you do, I would love your feedback.

Resources
This blog post stands on the shoulders of (and borrows from) the following blog posts by Maria Vexlard and Ram Vegiraju — big shout-outs to them: