SHAP: Explain Any Machine Learning Model in Python | by Louis Chan | Jan, 2023

By Jessie Hobb On Jan 12, 2023

Your Comprehensive Guide to SHAP, TreeSHAP, and DeepSHAP

Motivation

Story Time!

Imagine you have trained a machine learning model to predict the default risk of mortgage applicants. All is good, and the performance is excellent too. But how does the model work? How does the model come to the predicted value?

We stood there and said that the model considers several variables and the multi-dimensional relationship and pattern are too complex to be explained in plain words.

That’s where model explainability could save the day. Among the algorithms that can dissect machine learning models, SHAP is one of the more agnostic players in the field. In this blog, we will dive deep into the following items:

What are Shapley values?
How to calculate them?
How to use it in Python?
How does SHAP support local and global explanability?
What visualizations are available in the SHAP library?
How do the common variants of SHAP work? — TreeSHAP & DeepSHAP
How does LIME compare against SHAP?

Let’s Play a Game

When a team of eleven players goes on to win the World Cup, who is the most valuable player? Shapley value is a decomposition algorithm that objectively distributes the final result to a pool of factors. In explaining a machine learning model, Shapley values can be understood as the significance of individual input features’ contribution to the model’s predicted values.

A Quick Example — How does Shapley value work?

For simplicity’s sake, let’s say we have three attacking players, each with a different expected number of goals. We also know that these three players don’t always work well with each other, which means depending on the combination of the three players, the number of expected goals may be different:

As a baseline, we play none of these three players i.e. number of features f = 0 and the expected number of goals of the team will be 0.5. Each of the arrow that goes down the matrice indicates a possible stepwise increment when including a new feature (or including a player in our case).

Following the idea of stepwise expansion of player set, that means we can compute the marginal change for each of the arrow. For example, when we move from playing none of the players (indicated with the empty set symbol ∅) to playing player 1 only, the marginal change is:

Image by Author

To obtain the overall contribution of Player 1 among all three players, we would have to repeat the same calculation for every scenario where a marginal contribution for Player 1 is possible:

With all the marginal changes, we then calculate the weights for them using the following formula:

Image by Author

Or, to put it even simpler: it is just the reciprocal of the number of all edges pointing into the same row. That means:

With this, we can now calculate the SHAP value of Player 1 for the expected goals:

Repeating for the other two players and we will have:

SHAP of Player 1 = -0.1133
SHAP of Player 2 = -0.0233
SHAP of Player 3 = +0.4666

If I were the head coach, I would have only played Player 3 in this case.

This is very similar to another operator called Choquet Integral for those of you who are math-savvier.

Computational Complexity

With the above example of only 3 features, we would need to consider 8 different models, each with a different input feature set to explain all the features fully. In fact, for a full feature set of N features, the total number of subsets would be 2^N. Hence, we should be careful with the expected run time when using SHAP to explain machine learning models trained with a tall and, more importantly, wide dataset.

In the following sections, we will first dive into how we can use SHAP in Python before diverting most of our attention to different variants of SHAP that aim at tackling the complexity of SHAP either with approximation techniques or techniques that are model topology specific.

Next, let’s look at how to use SHAP in Python

SHAP (SHapley Additive exPlanations) is a python library compatible with most machine learning model topologies. Installing it is as simple as pip install shap.

SHAP provides two ways of explaining a machine learning model — global and local explainability.

Local Explainability with SHAP

Local explainability attempts to explain the driving forces behind a specific prediction. In SHAP, that’s what the individual Shapley values are used for, as illustrated in the quick example in an earlier section.

In SHAP’s arsenal, two visualizations are implemented to explain individual predictions: waterfall graph and force graph. While the waterfall graph gives you a better understanding of a stepwise derivation to the prediction results, the force graph is designed to provide a sense of the relative strength of the features’ contribution to the deviations in prediction results.

Note: Both visualizations included an overall expected prediction value (or base value). That can be understood as the average model output across the training set.

Waterfall Plot

# Code snippet from SHAP github page
import xgboost
import shap# train an XGBoost model
X, y = shap.datasets.boston()
model = xgboost.XGBRegressor().fit(X, y)
# explain the model's predictions using SHAP
# (same syntax works for LightGBM, CatBoost, scikit-learn, transformers, Spark, etc.)
explainer = shap.Explainer(model)
shap_values = explainer(X)
# visualize the first prediction's explanation
shap.plots.waterfall(shap_values[0])

Image from SHAP GitHub page (MIT license)

On the y-axis, you can find the feature’s name and value
On the x-axis, you can find the base value E[f(X)] = 22.533 that indicates the average predicted values across the training set
A red bar in this plot shows the feature’s positive contribution to the predicted value
A blue bar in this plot shows the feature’s negative contribution to the predicted value
The label on the bars indicates the deviation from the model’s base prediction value attributed to the parameter. For example, the AGE = 65.2 has marginally contributed +0.19 to the prediction’s deviation from the base value of 22.533
The bars are in descending order of the absolute importance of its impact on the predicted value

Force Plot

# Code snippet from SHAP github page
# visualize the first prediction's explanation with a force plot
shap.plots.force(shap_values[0])

On the x-axis, you can find the base value. That indicates the approximate location of the average predicted values across the training set.
On the x-axis, you can also find the model output with a bolded numeral. That indicates the predicted value for this record.
At the bottom of the chart, you can find the feature’s name and value, labelled either in red or blue.
All the red bars on the left of the model output are the features that have contributed positively to the prediction’s deviation from the base value. The names of the feature are at the bottom of the bars. The length of the bar indicates the features’ contributions.
All the blue bars on the right of the model output are the features that have contributed negatively to the prediction’s deviation from the base value. The names of the feature are at the bottom of the bars. The length of the bar indicates the features’ contributions.

Global Explainability with SHAP

Global explainability can be understood as understanding the overall importance of each feature in the model across the entire dataset and providing a general knowledge of the data and the underlying patterns. Due to the fuzziness in decomposing individual predictions’ contributions and aggregating across the data, there is more than one way to attempt global explainability. Examples include information gain, aggregated weights, permutation-based feature importance, and Shapley values. SHAP focuses on the last one, of course.

SHAP provides a visualization in which we can look into the average Shapley values of a feature across the dataset. Unlike other mechanisms that provide a measure of importance using statistically more complex interpretations, SHAP’s global explainability delivers an immediately understandable impact by allowing you to say that, on average, the feature relationship pushes the prediction value about 1.0 higher for data records with “Class 1” than data records with “Class 0”.

SHAP’s global explainability feature allows us to troubleshoot or investigate model bias. Taking the image above as an example, Age is generally a very significant feature. Could this be a sign that the model is biased towards specific age groups unnecessarily? Also, could one of the very important features be a potential data leak? All these questions allow us to improve the model before deploying a more responsible and robust machine-learning model.

Note: If you are interested in learning more about responsible AI, I have also written a piece on how we can approach that using 5 easy steps.

Another visualization that SHAP supports is a stacked version of the force graph in the local explainability section. By stacking the force charts, we can visualise the interactions between the model and the features that are given different input values. This gives us a clustering view based on Shapley values and provides us with perspectives on how the model sees the data. This can be very powerful for revising and validating hypotheses and underlying business logic. There might also be chances that you would find new ways of segregating your data after analysing all the Shapley values!

# Code snippet from SHAP github page
# visualize all the training set predictions
shap.plots.force(shap_values)

TreeSHAP

Pros: Efficient and accurate algorithm for computing Shapley values of tree-based models.
Cons: Only applicable to tree-based models.

Unlike the original SHAP, TreeSHAP is tree-based machine learning model-specific. This means TreeSHAP will only work on models such as decision trees, random forests, gradient-boosting machines etc.

TreeSHAP is specific to tree models because it takes advantage of the tree structures for computing accurate Shapley values more efficiently than SHAP. As these structures do not exist in other model topologies, TreeSHAP is only restricted to tree-based models.

TreeSHAP can calculate Shapley values using interventional and tree path dependent approaches. This can be specified in the feature_perturbation parameter. The tree path dependent approach calculates the changes in conditional expectation recursively. Let’s use a simple decision tree that accepts 2 features (x, y) as an example:

In the example above, we have a decision tree that contains 7 nodes, accepts two features (x, y) to predict z and has been trained with 8 training samples. To compute the local contribution of y to the prediction of z in a coalition (x=10, y=5), we need to consider the following:

For (x=10, y=5), the model will go from Node 1 to Node 3 and reach Node 6. As Node 6 is a leaf node, the model is certain that the prediction is z=4.
For (x=10), the model will go from Node 1 to Node 3. However, as Node 3 is not a leave node, the expected predicted value can be inferred as a weighted sum of all the leaf nodes of Node 3. Among the 5 training samples that went through Node 3, two are predicted to have z=4 while the others are predicted to have z=24. The weighted sum is 4*(2/5) + 24*(3/5)=1.6 + 14.4 = 16.
The marginal contribution of y in the prediction of z for the coalition (x=10, y=5) can be calculated as Prediction(x=10, y=5) — Prediction(x=10) = 4–16= -12.

Note: The negative contribution here does not mean the feature y is unimportant, but rather that feature y has pushed the prediction value by -12.

By continuing the process across all the features, TreeSHAP will obtain all the Shapley values and provide both local explainability (using the method above) and global explainability (average out all the local explainability results across the training set)

As its name suggests, the interventional approach calculates the Shapley values by artificially adjusting the value of the feature of interest. In our case above, that can be to change y from 5 to 4. To estimate the sensitivity, TreeSHAP will need to repeatedly use a background set/training set as reference points (This will be touched on again when we discuss LIME in the final section) with linear runtime complexity. Hence, when using the interventional approach, we should be more mindful of the scalability of TreeSHAP.

import shap# Load the data
X_train, y_train, X_test, y_test = load_data()
# Train the model
model = MyModel.train(X_train, y_train)
# Explain the model's predictions using the tree path dependent approach
explainer = shap.TreeExplainer(
model,
X_train,
feature_perturbation='tree_path_dependent')
shap_values_path = explainer.shap_values(X_test)
# Display the explanations
shap.summary_plot(shap_values_path, X_test)
# Explain the model's predictions using the interventional approach
explainer = shap.TreeExplainer(
model,
X_train,
feature_perturbation='interventional')
shap_values_interv = explainer.shap_values(X_test)
# Display the explanations
shap.summary_plot(shap_values_interv, X_test)

DeepSHAP

Pros: Efficient algorithm for approximating Shapley values of deep learning or neural network based models. Compatible with Tensorflow and PyTorch
Cons: Only applicable to deep learning or neural network based models. Less accurate than SHAP due to the approximation nature of the algorithm

We can’t skip neural networks when discussing explainability. DeepSHAP is a combination of SHAP and DeepLIFT that aims at cracking the philosophy behind deep learning models. It is specifically designed for deep learning models, which makes DeepSHAP only applicable to neural network based models.

DeepSHAP tries to approximate the Shapley values. A relatively primitive way of explaining DeepSHAP is that it attempts to assign local marginal contribution of feature x using gradients or partial derivatives with a meaningful background/reference point (e.g. pitch black for image recognition models, 0% for predicting one’s chance to get-rich-quick).

Note: There is a further research released on a generalised version of DeepSHAP — G-DeepSHAP. Feel free to give it a read here in arxiv.

import shap# Load the data
X_train, y_train, X_test, y_test = load_data()
# Train the model
model = MyModel.train(X_train, y_train)
# Explain the model's predictions using TreeSHAP
explainer = shap.DeepExplainer(model, X_train)
shap_values = explainer.shap_values(X_test)
# Display the explanations
shap.summary_plot(shap_values, X_test)

LIME — Alternative to SHAP

LIME(Local Interpretable Model-Agnostic Explanations) is an alternative to SHAP for explaining predictions. It is a model-agnostic approach with a default assumption in kernel size (size of local neighbourhood considered when explaining individual prediction) for approximating a feature’s contribution to a local instance. In general, choosing a smaller kernel size, the results provided by LIME will lean towards local interpretation of how the values of the features have contributed to the prediction. (i.e. larger kernel size tends to provide a more global view)

However, the choice of kernel size should be carefully decided dependent on the data and the pattern. Hence, when using LIME, we should consider adjusting the kernel size accordingly to obtain a reasonable interpretation of the machine learning model.

To give it a try, we can install and use the package with:

pip install lime

import lime
import lime.lime_tabular# Load the data
X_train, y_train, X_test, y_test = load_data()
feature_names = X_train.columns
# Train the model
model = MyModel.train(X_train, y_train)
# Explain the model's predictions using LIME
explainer = lime.lime_tabular.LimeTabularExplainer(
X_train, feature_names=feature_names)
# Choose a kernel size for the local neighborhood
kernel_size = 10
# Explain the model's prediction for a single instance
instance = X_test[0]
exp = explainer.explain_instance(
instance,
model.predict,
num_features=10,
kernel_size=kernel_size)
# Display the explanations
exp.show_in_notebook(show_all=False)

Conclusion

As a final recap, here is a quick summary of everything discussed in this blog post:

SHAP is a game theory based approach for explaining machine learning models
SHAP considers all possible combinations of features to evaluate the impact of every feature
SHAP value of a feature f for a local prediction instance is a weighted sum of the marginal changes due to the inclusion of the feature across all the possible combination of features that includes f
The marginal changes are weighted according to the reciprocal of f × C(F, f) for F to be the number of features considered by the actual model and f to be the number of features considered when calculating the marginal changes
As SHAP considers all possible combinations of features, hence the algorithm does not scale linearly and would suffer from the curse of dimensionality
Several variants of SHAP have been commonly used to address SHAP’s computational complexity:

We should consider using TreeSHAP for tree-based models and DeepSHAP for deep learning based models
LIME is an alternative to SHAP that is also model agnostic that approximates a feature’s contribution
Explanation by LIME can be significantly different depending on the choice of kernel size

Your Comprehensive Guide to SHAP, TreeSHAP, and DeepSHAP

Motivation

Story Time!

We stood there and said that the model considers several variables and the multi-dimensional relationship and pattern are too complex to be explained in plain words.

What are Shapley values?
How to calculate them?
How to use it in Python?
How does SHAP support local and global explanability?
What visualizations are available in the SHAP library?
How do the common variants of SHAP work? — TreeSHAP & DeepSHAP
How does LIME compare against SHAP?

Let’s Play a Game

A Quick Example — How does Shapley value work?

Image by Author

To obtain the overall contribution of Player 1 among all three players, we would have to repeat the same calculation for every scenario where a marginal contribution for Player 1 is possible:

With all the marginal changes, we then calculate the weights for them using the following formula:

Image by Author

Or, to put it even simpler: it is just the reciprocal of the number of all edges pointing into the same row. That means:

With this, we can now calculate the SHAP value of Player 1 for the expected goals:

Repeating for the other two players and we will have:

SHAP of Player 1 = -0.1133
SHAP of Player 2 = -0.0233
SHAP of Player 3 = +0.4666

If I were the head coach, I would have only played Player 3 in this case.

This is very similar to another operator called Choquet Integral for those of you who are math-savvier.

Computational Complexity

Next, let’s look at how to use SHAP in Python

SHAP (SHapley Additive exPlanations) is a python library compatible with most machine learning model topologies. Installing it is as simple as pip install shap.

SHAP provides two ways of explaining a machine learning model — global and local explainability.

Local Explainability with SHAP

Note: Both visualizations included an overall expected prediction value (or base value). That can be understood as the average model output across the training set.

Waterfall Plot

# Code snippet from SHAP github page
import xgboost
import shap# train an XGBoost model
X, y = shap.datasets.boston()
model = xgboost.XGBRegressor().fit(X, y)
# explain the model's predictions using SHAP
# (same syntax works for LightGBM, CatBoost, scikit-learn, transformers, Spark, etc.)
explainer = shap.Explainer(model)
shap_values = explainer(X)
# visualize the first prediction's explanation
shap.plots.waterfall(shap_values[0])

On the y-axis, you can find the feature’s name and value
On the x-axis, you can find the base value E[f(X)] = 22.533 that indicates the average predicted values across the training set
A red bar in this plot shows the feature’s positive contribution to the predicted value
A blue bar in this plot shows the feature’s negative contribution to the predicted value
The label on the bars indicates the deviation from the model’s base prediction value attributed to the parameter. For example, the AGE = 65.2 has marginally contributed +0.19 to the prediction’s deviation from the base value of 22.533
The bars are in descending order of the absolute importance of its impact on the predicted value

Force Plot

# Code snippet from SHAP github page
# visualize the first prediction's explanation with a force plot
shap.plots.force(shap_values[0])

On the x-axis, you can find the base value. That indicates the approximate location of the average predicted values across the training set.
On the x-axis, you can also find the model output with a bolded numeral. That indicates the predicted value for this record.
At the bottom of the chart, you can find the feature’s name and value, labelled either in red or blue.
All the red bars on the left of the model output are the features that have contributed positively to the prediction’s deviation from the base value. The names of the feature are at the bottom of the bars. The length of the bar indicates the features’ contributions.
All the blue bars on the right of the model output are the features that have contributed negatively to the prediction’s deviation from the base value. The names of the feature are at the bottom of the bars. The length of the bar indicates the features’ contributions.

Global Explainability with SHAP

Note: If you are interested in learning more about responsible AI, I have also written a piece on how we can approach that using 5 easy steps.

# Code snippet from SHAP github page
# visualize all the training set predictions
shap.plots.force(shap_values)

TreeSHAP

Pros: Efficient and accurate algorithm for computing Shapley values of tree-based models.
Cons: Only applicable to tree-based models.

Unlike the original SHAP, TreeSHAP is tree-based machine learning model-specific. This means TreeSHAP will only work on models such as decision trees, random forests, gradient-boosting machines etc.

For (x=10, y=5), the model will go from Node 1 to Node 3 and reach Node 6. As Node 6 is a leaf node, the model is certain that the prediction is z=4.
For (x=10), the model will go from Node 1 to Node 3. However, as Node 3 is not a leave node, the expected predicted value can be inferred as a weighted sum of all the leaf nodes of Node 3. Among the 5 training samples that went through Node 3, two are predicted to have z=4 while the others are predicted to have z=24. The weighted sum is 4*(2/5) + 24*(3/5)=1.6 + 14.4 = 16.
The marginal contribution of y in the prediction of z for the coalition (x=10, y=5) can be calculated as Prediction(x=10, y=5) — Prediction(x=10) = 4–16= -12.

Note: The negative contribution here does not mean the feature y is unimportant, but rather that feature y has pushed the prediction value by -12.

import shap# Load the data
X_train, y_train, X_test, y_test = load_data()
# Train the model
model = MyModel.train(X_train, y_train)
# Explain the model's predictions using the tree path dependent approach
explainer = shap.TreeExplainer(
model,
X_train,
feature_perturbation='tree_path_dependent')
shap_values_path = explainer.shap_values(X_test)
# Display the explanations
shap.summary_plot(shap_values_path, X_test)
# Explain the model's predictions using the interventional approach
explainer = shap.TreeExplainer(
model,
X_train,
feature_perturbation='interventional')
shap_values_interv = explainer.shap_values(X_test)
# Display the explanations
shap.summary_plot(shap_values_interv, X_test)

DeepSHAP

Pros: Efficient algorithm for approximating Shapley values of deep learning or neural network based models. Compatible with Tensorflow and PyTorch
Cons: Only applicable to deep learning or neural network based models. Less accurate than SHAP due to the approximation nature of the algorithm

Note: There is a further research released on a generalised version of DeepSHAP — G-DeepSHAP. Feel free to give it a read here in arxiv.

import shap# Load the data
X_train, y_train, X_test, y_test = load_data()
# Train the model
model = MyModel.train(X_train, y_train)
# Explain the model's predictions using TreeSHAP
explainer = shap.DeepExplainer(model, X_train)
shap_values = explainer.shap_values(X_test)
# Display the explanations
shap.summary_plot(shap_values, X_test)

LIME — Alternative to SHAP

To give it a try, we can install and use the package with:

pip install lime

import lime
import lime.lime_tabular# Load the data
X_train, y_train, X_test, y_test = load_data()
feature_names = X_train.columns
# Train the model
model = MyModel.train(X_train, y_train)
# Explain the model's predictions using LIME
explainer = lime.lime_tabular.LimeTabularExplainer(
X_train, feature_names=feature_names)
# Choose a kernel size for the local neighborhood
kernel_size = 10
# Explain the model's prediction for a single instance
instance = X_test[0]
exp = explainer.explain_instance(
instance,
model.predict,
num_features=10,
kernel_size=kernel_size)
# Display the explanations
exp.show_in_notebook(show_all=False)

Conclusion

As a final recap, here is a quick summary of everything discussed in this blog post:

SHAP is a game theory based approach for explaining machine learning models
SHAP considers all possible combinations of features to evaluate the impact of every feature
SHAP value of a feature f for a local prediction instance is a weighted sum of the marginal changes due to the inclusion of the feature across all the possible combination of features that includes f
The marginal changes are weighted according to the reciprocal of f × C(F, f) for F to be the number of features considered by the actual model and f to be the number of features considered when calculating the marginal changes
As SHAP considers all possible combinations of features, hence the algorithm does not scale linearly and would suffer from the curse of dimensionality
Several variants of SHAP have been commonly used to address SHAP’s computational complexity:

We should consider using TreeSHAP for tree-based models and DeepSHAP for deep learning based models
LIME is an alternative to SHAP that is also model agnostic that approximates a feature’s contribution
Explanation by LIME can be significantly different depending on the choice of kernel size

Read original article here

Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.