Explainable AI: Unfold the Blackbox | by Charu Makhijani | May, 2022

By Jessie Hobb On May 27, 2022

Build trust in machine learning with XAI, Guide to SHAP & SHapley Values

With the advent of AI becoming more advanced and becoming a crucial part of our lives, the danger comes when we don’t understand the effects and side effects of AI. It is important to understand how to differentiate between the fact and fantasies of the AI decision-making process while retaining the AI efficiencies and providing the most transparency with the outcomes. All this can be achieved with Explainable AI (XAI).

Explainable AI is a technique that aims to create human-interpretable machine learning. The techniques that will help humans understand, interpret and trust predictions made by the machine learning models.

The image above provides a representation of Regular vs Explainable AI and how it can affect the end-users.

When we understand the logic behind the predictions from an ML algorithm, it is no longer a black box. This will not only help the data scientists & ML engineers to explain the story behind the predictions, but it will also help the business and overall organization to trust and adopt AI.

Explainable AI Is more important in high stake AI domains like- Financial Investments, Medical diagnoses, Autonomous Vehicles, and Legal or Defense-related decision-making. Currently, it’s hard to trust the inference of the ‘Black Box’ AI models in these places.

Benefits

Building Confidence in AI-driven Business Decisions- Explainable AI will help in creating trust & confidence in business decisions. Inference from Explainable AI systems tends to increase the system’s confidence, as it’s possible to see the main drivers of these decisions.
Transparent and Accountable AI Systems- As AI takes on projects with increasing impact, explainability will make sure the research and decision-making are transparent. Hence it will be easier to make engineers/teams/organizations to be accountable for the decisions.
Mitigate legal, compliance & reputational risks- Using AI is not enough, there is an obligation to use it fairly. Keeping the AI models explainable and transparent, can greatly reduce the impact of erroneous results and organizations can mitigate the risks from regulatory & compliance bodies.
Mitigate Biases & promote fairness- With Explainable AI, it’s easier to see the patterns in the data that influence the outputs of ML models. Hence explainable systems can reduce the impact of biased predictions, mitigate their risk and foster fairness & trust in the predictions.

While creating an ML model, there is always a trade-off between model accuracy/complexity vs model explainability.

Most of the simple predictive models such as linear models are less accurate, less complex and can be easily explained. Whereas advanced methods such as Neural Networks are more accurate, capable of making predictions on more complex data and are not easily explainable. Hence balancing explainability and model complexity is always a challenge. When model complexity increases, explainability decreases.

The figure below shows the tradeoff between model complexity and explainability for different ML models.

Explainable AI can play a vital role in increasing the explainability of highly complex models. The models that are not self-explanatory can be interfaced with XAI methods that can provide feature importance-based insights based on local and global explanations. There are many open-source frameworks and techniques that can be used to explain complex ML methods to non-AI expert people. Let’s see the techniques.

XAI techniques can be divided into two categories-

Transparent Methods
Post-Hoc methods

Transparent Methods are where the intrinsic architecture of the model is simple to interpret and feature correlation is not complex. Linear/Logistic Regression, Decision Trees, KNN and Bayesian models are examples of transparent models.

Post-Hoc methods are useful where there are complex decision boundaries in the data and model complexity is hard to interpret. Post-Hoc Methods are further divided into two categories-

Model Specific- Model-specific methods work on a single or a group of models and rely on the functionality and capabilities of a specific model type.
Model Agnostic- Model Agnostic methods work on any ML model by analyzing model input & output features and do not rely on model functionality or complexity. These methods are useful when there is no way to interpret model functionality.

The figure below shows the various XAI techniques-

The figure above shows only one way of looking at different types of XAI methods. There are other ways to classify XAI based on

Scope- Global or local
Access- The explanation method has limited or full access to the model
Explanation stage- Explanations are applied during model development (training) or after training

XAI is vast and there are many methods. In this post, I will cover only the SHAP method. The rest of the model agnostic and model-specific techniques will be covered in the next part of the XAI posts.

SHAP (SHapley Additive exPlanations) method is a game-theoretic approach in python to explain the output of any machine learning model using Shapley values.

To understand SHAP let’s first learn what are Shapley Values.

Shapley Values

The Shapley value is a concept from game theory that measures the average marginal contribution of a feature instance in the prediction.

Let’s first understand why Shapley values are important. Take an example of Income Prediction where we have to classify if income is ≤50K or > 50K. We have used xgboost classifier to predict the income(you can find the complete code here). Now we have to check what features have contributed to the prediction, we can check this using the plot_importance method. This plot_importance method has a parameter called importance_type that calculates the feature importance based on weight (number of times feature appears in a tree), gain (average gain of splits using the feature) and cover (average coverage of splits using the feature).

Looking at these 3 values of importance_type, we get 3 different sets of feature contributions using Xgboost, but we don’t know which one method is best and how to get the reliable features that actually contributed to the model. The second issue using this measure is that this method can contribute only to global interpretability, and won’t work if we have to get the important features for a specific record (local interpretability).

SHAP method in python can help us solve both of these problems. But before looking into the SHAP implementation and how these problems can be resolved, first let’s understand what are Shapley values and how they are calculated with the same Income Prediction example.

While SHAP method is useful to get the importance of feature for model prediction, it is not a measure to evaluate the prediction itself.

To predict if income is > 50K or not, suppose there is the contribution of these 3 features — age, marital status and capital gain. Now let’s understand the marginal contribution of these features. As shown in the figure below, here-

age contributes: $25K
marital-status contributes: $45K
capital-gain contributes: $70K
age and marital-status contribute: $32K
age and capital-gain contribute: $60K
capital-gain and marital-status contribute: $75K
age, marital-status and capital-gain contribute: $95K

To find the marginal contribution of Age, we have to calculate the marginal contribution of Age with-

$50K Income group
Age and Marital Status group
Age and Capital Gain group
Age, Marital Status and Capital Gain group

This is shown in the figure below. In the same way, we can calculate the marginal contribution of all features.

Now let’s calculate the weights for each level.

The 1st level has 3 edges so the weight of each edge on this level will be 1/3 — let’s call it w1
The 2nd level has 6 edges so the weight of each edge on this level will be 1/6 — let’s call it w2
The 3rd level has 3 edges so the weight of each edge on this level will be 1/3 — let’s call it w3

Now we have the marginal contribution of features and weight for all levels, let’s calculate the total contribution of features.

This shows Age contributes -$7.5K to the Income Prediction, so -$7.5K is the Shapley value for Age.

In the same way, we can calculate Shapley Values for Marital Status and Capital Gain.

SHAP(age) = -$7.5K

SHAP(marital status) = $11.5K

SHAP(capital gain) = $38K

If we sum Shapley values from these 3 features it’s $42K (-$7.5K+$11.5K+$38K), which is equal to a path from $50K to $92K income ($92K-$50K).

Now as we understand what are Shapley values and how to calculate them, let’s see the practical way of implementing using the SHAP library in python.

To use SHAP, first pip install the SHAP library and import it. Then create a SHAP explainer object and use it to get the shap values to explain the model’s predictions.

Now let’s understand the global and local interpretability using Shapley Values.

Global Interpretability

Summary Bar Plot- This plot shows the feature importance in descending order. The features on top of the plot have more contribution to model predictions and the features on the bottom of the plot have lesser contributions.

Summary Plot- The summary plot also shows feature importance in descending order, plus it shows the impact and correlation of each feature with the target variable. For each observation, we can see the high/low impact with the color turning from Blue(low) to Red(high). We can see the correlation as high capital-gain (red) has a positive impact (moving to the +ve axis) on income, whereas age has a negative correlation with income.

Dependence Plot- The dependence plot shows the effect of a feature on the model’s predictions. The y-axis variable is selected automatically, most often this variable is the one where the feature interacts more often for making model predictions.

We can also change the y-axis variable and change the feature interaction plot by using parameter interaction_index and providing another variable for the y-axis.

Force Plot- The force plot shows how features contribute to the model output. Features contributing more are in red, and features contributing less are in blue.

Local Interpretability

Bar plot- This plot shows the contribution of features in that specific observation. The below plot shows that capital-loss, relationship and education are the most contributing features to this observation.

Waterfall Plot- The Waterflow plot is another way of plotting the contribution of a feature in a specific observation as shown below.

Force Plot- For a specific observation, force plot shows what features have most contribution.

Using SHAP, we are not only getting interpretability at the global and local levels, but the feature set is also not changing with different plots. How cool is that!!

We have seen above how XAI can help us get the Why and How answers about the AI systems. As AI is becoming more mainstream in many industries, so does the need for explainability. However, despite the growing interest in Explainable AI, there is a big gap between the Explainable AI Vision and practice. There can be multiple explanations required from the same AI algorithm, so how Explainable AI will help in this context is still a question. Also, there is a lack of guidance from real-world use cases to implement & test the explanations.

Finally, XAI is a new and developing field, there are many challenges and open questions, but there is no doubt that XAI will play an integral part in the future of the AI literature.

To access the complete code for income Prediction using Xgboost and explain it using SHAP, please refer GitHub link.