Machine Learning, Illustrated: Opening Black Box Models with SHAP | by Shreya Rao | May, 2023

By Jessie Hobb On May 8, 2023

How to explain any machine learning model using SHAP

Shapley Value is a concept derived from cooperative game theory in Economics that assigns a value to each player in a cooperative game based on their contributions to the game. In the field of machine learning, this concept has been adapted into the SHAP (SHapley Additive exPlanations) framework, which is an effective technique for interpreting the workings of a model.

If you’re interested in learning more about Shapley Values, I highly recommend checking out my previous article on the math and intuition behind Shapley Values. Even though it’s been modified for machine learning purposes, understanding its underlying principles can be helpful.

The SHAP framework is similar to Shapley Values in that it calculates the individual impact of features in a game (aka a machine learning model). However, machine learning models are non-cooperative games, which means that features don’t necessarily interact with each other as they do in cooperative games. Instead, each feature contributes independently to the model’s output. While the Shapley Values formula can be used, it can be computationally expensive and inaccurate due to the large number of players and coalitions involved. To solve this problem, researchers have developed alternative methods such as the Monte Carlo method and kernel-based methods. In this article, we will delve deeper into the Monte Carlo method.

Let’s set this up with an example. Say we have a dataset of the prices of 20,640 houses in California.

from sklearn.datasets import fetch_california_housing# load dataset
housing = fetch_california_housing()
X, y = housing.data, housing.target
X = pd.DataFrame(X, columns=housing.feature_names)
# feature dataset
X = X.drop(['Population', 'AveBedrms', 'AveOccup'], axis=1)

We will use the features MedInc, HouseAge, AveRooms, Latitude, and Longitude (X)…

…to predict the price of a house (y). Note: The house prices are expressed in $100,000.

Let’s build a model. This can be any model but we’ll use XGBoost. Following the standard steps — divide the data into train and test sets and train the model with the train set.

# split data into train and test
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)# train an XGBoost model
from xgboost import XGBRegressor
model = xgb.XGBRegressor(objective='reg:squarederror', random_state=4)
model.fit(X_train, y_train)

Then use this model to make predictions on the test set.

y_pred = model.predict(X_test)

Let’s break this down further. We want the predicted house price of the first sample (or house) in our test set. So for the first sample with these feature values…

X_test.iloc[0,]

MedInc         4.151800
HouseAge      22.000000
AveRooms       5.663073
Latitude      32.580000
Longitude   -117.050000

…the predicted house price is:

y_pred[0]
> 1.596

The prediction made by XGBoost seems mysterious as it is a black-box model so we don’t see what’s happening inside the box. To gain more insight into how the value was predicted, there are two things we need to understand:

Do the features affect the predicted house price positively or negatively? (For instance, does a higher MedInc value lead to an increase in the predicted house price?)
What is the extent of these influences? (For instance, does MedInc have a more significant impact on the predicted house price than AveRooms?)

To answer these questions and determine the contribution of each feature to the initial house price prediction of 1.596, we can rely on the SHAP values of the features.

Let’s start by calculating the SHAP value of MedInc to this prediction of 1.596 by following these steps:

Step 0: Calculate the expected predicted house price

The expected predicted house price is nothing but the average of all the predictions made. Which is:

y_pred.mean()
> 2.07

So, 2.07 serves as the expected predicted house price. We are now trying to figure out why the predicted value of 1.596 deviates from the expected prediction of 2.07. Where is the discrepancy of 0.476 (2.07–1.596) coming from?

The SHAP value of a feature measures how much that feature contributed to moving the model’s prediction away/towards the expected predicted value.

Step 1: Get a random permutation of the features

We started with this permutation of features…

…and after a random reshuffle, we get this:

We care about MedInc here, so let’s highlight MedInc and any feature to the right of it.

Step 2: Pick a random sample from the dataset

We go back to our test dataset and choose a random sample from it:

Step 3: Form two new samples

These samples are going to be formed partially from the original sample…

…and the new sample that we just pulled in Step 2:

The first sample that we create, called x1, is going to have all the same values from our original sample except for the features to the right of MedInc. For the features to the right of MedInc, we get from the new sample.

The second sample that we create, called x2, is going to have all the values from the original sample except for the features to the right of MedInc and MedInc.

From this, we can see that these are the exact sample samples except for one key thing — they vary in their MedInc value, the feature we care about.

Step 4: Use the new samples to make predictions and find the difference in predictions

Now we run these samples through our model and get predictions for them.

And we find the difference between these values:

Since the only difference between x1 and x2 is the value of MedInc, this difference represents the change in prediction when the value of MedInc is altered.

Step 5: Repeat Steps 1–4 a couple of times and then calculate the average of the difference

All we do now is repeat the above process and calculate the average of all the difference values found in Step 4.

Let’s assume that we repeat this process 1000 times (this number may vary based on the complexity of the model and dataset). After averaging the results, we obtained a value of 0.22.

This 0.22 represents the SHAP value of MedInc in the first sample which implies that MedInc contributes +0.22 towards adjusting the expected prediction of 2.07 to our prediction of 1.596.

We can apply the same process to the other features — HouseAge, AveRooms, Latitude, and Longitude to find their SHAP values.

Now that we understand the underlying calculations of SHAP, we can apply it to our predictions by visualizing them. To visualize them, we will use Explainer from Python’s shap library and input our model.

import shap
explainer = shap.Explainer(model)
shap_values = explainer(X_test)

This will give us the SHAP values for all the features (MedInc, HouseAge, AveRooms, Latitude, and Longitude) for each sample in the test set. Using these SHAP values, let’s get plotting.

1. Waterfall Plot

This plot helps us visualize the SHAP values of each sample in our data individually. Let’s visualize the SHAP values of the first sample.

# visualize the SHAP values of the first sample
shap.plots.waterfall(shap_values[0])

Notice that the expected predicted value, E[f(X)] = 2.07, which is the value calculated in Step 0, and the predicted house price of the first house, f(X) = 1.596 is our prediction of the first sample.

Observe that the SHAP value of MedInc is +0.22 (which we see in Step 5, the SHAP value of Longitude is -2.35, etc. If we add and subtract all the above SHAP values to and from 2.07, respectively, we arrive at the predicted value of 1.596 for the first house.

Ignoring the signs, the magnitude of the SHAP value for Longitude, 2.35, is greater than that of the other features. This implied that Longitude has the most significant impact on the prediction.

Just like we visualized the SHAP values of the first sample, we can visualize SHAP values for the second sample too.

# visualize the SHAP values of the second sample
shap.plots.waterfall(shap_values[0])

Comparing the SHAP values for the first and second houses in our test data, we observe significant differences. In the first house, Longitude had the most significant impact on the predicted price, while in the second house, MedInc has the most prominent influence.

NOTE: These differences in the SHAP values highlight the unique contributions of each feature to the model’s output for each sample. It’s essential to understand these contributions to build trust in the model’s decision-making process and ensure that it’s not biased or discriminatory.

2. Force Plot

Another way to visualize the above is a force plot.

shap.plots.force(shap_values[0])

3. Mean SHAP Plot

To determine which features are generally most important for our model’s predictions, we can use a bar plot of the mean SHAP values across all observations. Taking the mean of the absolute values ensures that positive and negative values do not cancel each other out.

shap.plots.bar(shap_values)

Each feature has a corresponding bar, with the height representing the mean SHAP value. For instance, in our plot, the feature with the largest mean SHAP value is Latitude, indicating that it has the most substantial impact on our model’s predictions. This information can help us understand which features are critical to the model’s decision-making process.

4. Beeswarm Plot

The beeswarm plot is a useful visualization to examine all of the SHAP values for each feature. The y-axis groups the SHAP values by feature, with the color of the points indicating the corresponding feature value. Typically, redder points represent higher feature values.

The beeswarm plot can help identify important relationships between features and the model’s predictions. In this plot, the features are ordered by their mean SHAP values.

shap.plots.beeswarm(shap_values)

By examining the SHAP values in the beeswarm plot, we can start to understand the nature of the relationships between the features and the predicted house price. For instance, for MedInc, we observe that SHAP values increase as the feature value increases. This suggests that higher values of MedInc contribute to higher predicted house prices.

In contrast, for the Latitude and Longitude, we notice the opposite trend, where higher feature values lead to lower SHAP values. This observation implies that higher Latitude and Longitude values are associated with lower predicted house prices.

5. Dependence Plots

To gain a deeper understanding of the relationships between individual features and their corresponding SHAP values, we can create dependence plots. A dependence plot is a scatter plot that shows the relationship between the SHAP value and the feature value for a single feature.

shap.plots.scatter(shap_values[:,"MedInc"])

shap.plots.scatter(shap_values[:,"Latitude"])

By analyzing dependence plots, we can confirm the observations made in the beeswarm plot. For instance, when we create a dependence plot for MedInc, we observe a positive relationship between MedInc values and SHAP values. In other words, higher MedInc values result in higher predicted house prices.

Overall, the dependence plots provide a more detailed understanding of the complex relationships between individual features and predicted house prices.

In conclusion, SHAP values give us a peek and insights into how each feature contributes to the model’s output. These insights can be further enhanced through the use of visualizations. We can leverage SHAP to make more informed decisions about feature selection, model improvement, and ultimately, achieve better performance in our machine learning applications.

As always, thank you for including me in your machine learning journey. Shoot me an email at [email protected] or connect with me on LinkedIn for comments, questions, and suggestions!