Follow this step-by-step guide to get immediate insight into your classifier performance and be notified when it drops
The moment you deliver predictions to your customers you want to know how your model behaves. No need to wait for true/false positives/negatives, by calculating the classification rate you can get performance insights of your model right away!
This article focuses on how to combine AWS services to calculate the classification rate and get notified when your model is drifting.
The classification rate is calculated as the sum of the valid predictions divided by the total number of predictions.
classification rate = valid predictions / total predictions
Monitoring the classification rate at inference time is crucial for:
- Knowing if it’s similar to the classification rate at training time.
- Detecting model performance degradation when the classification rate drops.
What makes a valid prediction?
A valid prediction is a probability above a certain threshold. For example, we can consider all predictions above the threshold of 95% valid and everything below invalid predictions.
If 90 out of 100 predictions are higher than the threshold of 95%, this means that 90% of the predictions are valid and the classification rate is 90/100=0,9.
You can play with the threshold to get the classification rate right and optimize the performance of your model.
Now that we know what valid predictions are and what is the classification rate, let’s use the power of AWS Cloudwatch to get insight into the classification rate and trigger an alarm when the classification rate drops.
At inference time log if the prediction is valid or not. Here is a Python snippet with a simple example:
# prediction_valid = True | False
logger.info( json.dumps({
“prediction_valid”: prediction_valid
}))
In Cloudwatch Logs you should see the “prediction_valid” appear:
A lot of AWS services like Lambda or Sagemaker are natively integrated with Cloudwatch, so logging will work out of the box if your role has the correct IAM policies.¹
If your service is not integrated with Cloudwatch out of the box, for example a pod running on EKS, look at this StackOverflow answer on how to log to Cloudwatch using boto3.
There is a reason why we dumped JSON data to Cloudwatch in the previous step. We will query JSON data in our Cloudwatch log groups via Cloudwatch Metric filters and keep track of valid and total predictions.
Valid Predictions
- Go to your Cloudwatch Log group > Metric Filters > Create metric filters.
Use the following “Filter pattern” to get all valid predictions:
{ $.prediction_valid IS TRUE }
More on filter pattern syntax here.
- Use the button “Test pattern” to test your Filter pattern against a log stream.
- Click Next button.
- Filter name “valid_predictions”.
- Metric namespace “classification_rate”.
- Metric name “valid_predictions”.
- Metric value “1”, like this we count each occurrence of the valid predictions.
- Click Next and click “Create metric filter”
We created a metric filter for valid predictions. In the same way, we will create one for the total predictions.
Total Predictions
In Step1, do exactly the same steps as for the valid predictions but use the following filter pattern to get all predictions:
{ $.prediction_valid IS TRUE || $.prediction_valid IS FALSE}
This will query all occurrences of prediction_valid, true or false.
In Step 2, do
- Filter Name “total_predictions”.
- Metric Namespace “classification_rate”.
- Metric Name “total_predictions”.
- Metric Value “1”.
After creating the total_predictions metric filter you should see the following:
Metric filters are only using data as of the moment of creation, historical data is ingored! You might need to wait for a few moments before the metrics have collected some data.
- Go to Cloudwatch > Alarms > All Alarms > Create Alarm.
- Select metric.
- Select the Namespace “classification_rate”.
- Select “Metrics with no dimension”.
- Select the checkbox “valid_predictions” and “total_predictions”.
- Go to the tab Graphed Metrics.
- Set the Statistic to Sum.
- Add math > Common > percentage.
- Make sure you divide the id of valid_predictions by the id of total_predictions.
In this example, the classification rate is 100*(m2/m1). Where m2 is the id of valid_predictions and m1 is the id of total_predictions. We multiply by 100 to get a percentage.
- Edit the label > classification_rate.
- Only select the classification_rate > click on the button “Select a single metric to continue”.
We have created a Cloudwatch Metric for the classification rate.
Next, we will attach an alarm to the classification rate that will be triggered when the classification rate drops below some threshold and we will receive an email.
- Set the Period to 1 hour.
- Set the Threshold Type to “Static”.
- Whenever the classification rate is … Lower/Equal than … 90.
We use a threshold value of 90, which means that the alarm is triggered when the classification rate drops below 90. Typically the threshold value is a bit lower than the classification rate you have at training time.
- Select additional configuration.
- Datapoints to alarm > set it to “1 out of 1”.
Setting the alarm to “1 out of 1” can get you spammed with messages. We do this in the tutorial to make it easy to trigger an alarm and get an email in our inbox.
To avoid getting spammed with emails, you can configure the alarm to be less “nervous” by triggering the alarm only when multiple periods pass the threshold.
For example, when you want to trigger an alarm when the threshold is breached for 3 consecutive periods, you set it to “3 out of 3”. Remember, in our example we have set the period to 1 hour which would mean that for 3 hours the threshold is breached.
Alternatively, you can play with these numbers to set it to “3 out of 5″ for example. This means that an alarm is triggered when 3 periods out of 5 the threshold is breached.
- Missing data treatment > set it to “Treat missing data as good (not breaching threshold)”.
If you do not have a steady stream of data you might want to consider setting it to Treat missing data as good (not breaching threshold), like this you won’t be spammed with messages whenever data is missing for a certain period.²
- Alarm State Trigger > Create new topic.
- Set the name of the SNS topic to “classification_rate_alarm”.
- Fill in an email address that will receive a message whenever the alarm is triggered.
- Click on “create topic”.
- You will receive an email to confirm your subscription. Make sure to click the link. You should see this:
- Go back to the AWS console and click Next.
- Set the Alarm name and give it a description.
- Click Next.
- In the preview screen click the button “Create alarm”.
In the overview, you should see the alarm, maybe with the state “Insufficient data”. If that’s the case don’t worry, after a while the alarm has processed some data and the state will turn to “OK”.
When you click on the alarm you should see the classification rate in blue and in red the threshold that will trigger the alarm when it gets breached.
To test the alarm you could go to Edit > set the threshold to 100 so that the alarm gets triggered. After a few moments, you should receive an email.
We tested the alarm, you can set back the threshold to the original value.
That’s it! Now you can monitor the performance of your classification model and get notified when it drops.
👋 Follow me on Medium, Linkedin, and Twitter to read more on ML engineering and ML pipelines.
Footnotes
[1] Set IAM policies to log to Cloudwatch https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/iam-identity-based-access-control-cwl.html
[2] How Alarms are triggered in AWS Cloudwatch https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html
Follow this step-by-step guide to get immediate insight into your classifier performance and be notified when it drops
The moment you deliver predictions to your customers you want to know how your model behaves. No need to wait for true/false positives/negatives, by calculating the classification rate you can get performance insights of your model right away!
This article focuses on how to combine AWS services to calculate the classification rate and get notified when your model is drifting.
The classification rate is calculated as the sum of the valid predictions divided by the total number of predictions.
classification rate = valid predictions / total predictions
Monitoring the classification rate at inference time is crucial for:
- Knowing if it’s similar to the classification rate at training time.
- Detecting model performance degradation when the classification rate drops.
What makes a valid prediction?
A valid prediction is a probability above a certain threshold. For example, we can consider all predictions above the threshold of 95% valid and everything below invalid predictions.
If 90 out of 100 predictions are higher than the threshold of 95%, this means that 90% of the predictions are valid and the classification rate is 90/100=0,9.
You can play with the threshold to get the classification rate right and optimize the performance of your model.
Now that we know what valid predictions are and what is the classification rate, let’s use the power of AWS Cloudwatch to get insight into the classification rate and trigger an alarm when the classification rate drops.
At inference time log if the prediction is valid or not. Here is a Python snippet with a simple example:
# prediction_valid = True | False
logger.info( json.dumps({
“prediction_valid”: prediction_valid
}))
In Cloudwatch Logs you should see the “prediction_valid” appear:
A lot of AWS services like Lambda or Sagemaker are natively integrated with Cloudwatch, so logging will work out of the box if your role has the correct IAM policies.¹
If your service is not integrated with Cloudwatch out of the box, for example a pod running on EKS, look at this StackOverflow answer on how to log to Cloudwatch using boto3.
There is a reason why we dumped JSON data to Cloudwatch in the previous step. We will query JSON data in our Cloudwatch log groups via Cloudwatch Metric filters and keep track of valid and total predictions.
Valid Predictions
- Go to your Cloudwatch Log group > Metric Filters > Create metric filters.
Use the following “Filter pattern” to get all valid predictions:
{ $.prediction_valid IS TRUE }
More on filter pattern syntax here.
- Use the button “Test pattern” to test your Filter pattern against a log stream.
- Click Next button.
- Filter name “valid_predictions”.
- Metric namespace “classification_rate”.
- Metric name “valid_predictions”.
- Metric value “1”, like this we count each occurrence of the valid predictions.
- Click Next and click “Create metric filter”
We created a metric filter for valid predictions. In the same way, we will create one for the total predictions.
Total Predictions
In Step1, do exactly the same steps as for the valid predictions but use the following filter pattern to get all predictions:
{ $.prediction_valid IS TRUE || $.prediction_valid IS FALSE}
This will query all occurrences of prediction_valid, true or false.
In Step 2, do
- Filter Name “total_predictions”.
- Metric Namespace “classification_rate”.
- Metric Name “total_predictions”.
- Metric Value “1”.
After creating the total_predictions metric filter you should see the following:
Metric filters are only using data as of the moment of creation, historical data is ingored! You might need to wait for a few moments before the metrics have collected some data.
- Go to Cloudwatch > Alarms > All Alarms > Create Alarm.
- Select metric.
- Select the Namespace “classification_rate”.
- Select “Metrics with no dimension”.
- Select the checkbox “valid_predictions” and “total_predictions”.
- Go to the tab Graphed Metrics.
- Set the Statistic to Sum.
- Add math > Common > percentage.
- Make sure you divide the id of valid_predictions by the id of total_predictions.
In this example, the classification rate is 100*(m2/m1). Where m2 is the id of valid_predictions and m1 is the id of total_predictions. We multiply by 100 to get a percentage.
- Edit the label > classification_rate.
- Only select the classification_rate > click on the button “Select a single metric to continue”.
We have created a Cloudwatch Metric for the classification rate.
Next, we will attach an alarm to the classification rate that will be triggered when the classification rate drops below some threshold and we will receive an email.
- Set the Period to 1 hour.
- Set the Threshold Type to “Static”.
- Whenever the classification rate is … Lower/Equal than … 90.
We use a threshold value of 90, which means that the alarm is triggered when the classification rate drops below 90. Typically the threshold value is a bit lower than the classification rate you have at training time.
- Select additional configuration.
- Datapoints to alarm > set it to “1 out of 1”.
Setting the alarm to “1 out of 1” can get you spammed with messages. We do this in the tutorial to make it easy to trigger an alarm and get an email in our inbox.
To avoid getting spammed with emails, you can configure the alarm to be less “nervous” by triggering the alarm only when multiple periods pass the threshold.
For example, when you want to trigger an alarm when the threshold is breached for 3 consecutive periods, you set it to “3 out of 3”. Remember, in our example we have set the period to 1 hour which would mean that for 3 hours the threshold is breached.
Alternatively, you can play with these numbers to set it to “3 out of 5″ for example. This means that an alarm is triggered when 3 periods out of 5 the threshold is breached.
- Missing data treatment > set it to “Treat missing data as good (not breaching threshold)”.
If you do not have a steady stream of data you might want to consider setting it to Treat missing data as good (not breaching threshold), like this you won’t be spammed with messages whenever data is missing for a certain period.²
- Alarm State Trigger > Create new topic.
- Set the name of the SNS topic to “classification_rate_alarm”.
- Fill in an email address that will receive a message whenever the alarm is triggered.
- Click on “create topic”.
- You will receive an email to confirm your subscription. Make sure to click the link. You should see this:
- Go back to the AWS console and click Next.
- Set the Alarm name and give it a description.
- Click Next.
- In the preview screen click the button “Create alarm”.
In the overview, you should see the alarm, maybe with the state “Insufficient data”. If that’s the case don’t worry, after a while the alarm has processed some data and the state will turn to “OK”.
When you click on the alarm you should see the classification rate in blue and in red the threshold that will trigger the alarm when it gets breached.
To test the alarm you could go to Edit > set the threshold to 100 so that the alarm gets triggered. After a few moments, you should receive an email.
We tested the alarm, you can set back the threshold to the original value.
That’s it! Now you can monitor the performance of your classification model and get notified when it drops.
👋 Follow me on Medium, Linkedin, and Twitter to read more on ML engineering and ML pipelines.
Footnotes
[1] Set IAM policies to log to Cloudwatch https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/iam-identity-based-access-control-cwl.html
[2] How Alarms are triggered in AWS Cloudwatch https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html