The Art of Explaining Predictions | by Conor O’Sullivan | May, 2022
How to explain your model in a human-friendly way
An important part of a data scientist’s role is to explain model predictions. Often, the person receiving the explanation will be non-technical. If you start talking about cost functions, hyperparameters or p-values you will be met with blank stares. We need to translate these technical concepts into layman’s terms. This process can be more challenging than building the model itself.
We will explore how you can give human-friendly explanations. We will do this by discussing some key characteristics of a good explanation. The focus will be on explaining individual predictions. In the end, we will apply some of this knowledge by explaining a model using SHAP values. We will see that SHAP is very useful when you want to give human-friendly explanations.
Before we dive into that, let’s discuss what you’re explaining and who you’re explaining it to. As a data scientist, you will likely need to communicate with a variety of people. This includes colleagues, regulators or customers. These people will all have various levels of technical expertise. So you will need to adjust the level of your explanation based on their expertise.
In some cases, you could be asked to explain the model as a whole. We refer to this as a global explanation. We want to understand what trends are being captured by the model in general. We may need to answer questions like “which features are the most important?” or “what relationship does feature X have with the target variable?”
In other cases, we will need to give local explanations. This is when we explain individual model predictions. In fact, we will usually have to explain a decision that has resulted from a model prediction. These explanations can also be answers to questions like “why did we reject this loan application?” or “why was I given this movie recommendation?”
When talking to a colleague or regulator you may need to give more technical explanations. In comparison, customers would expect simpler explanations. It is also unlikely that you would need to give a global explanation to a customer. This is because they would typically only be concerned with decisions that affect them personally. We will focus on this scenario. That is explaining individual predictions to a non-technical person.
When we talk about a good explanation we mean one that will be readily accepted. In other words, it should convince the audience that a decision was correct. To give such an explanation there are some key aspects you need to consider. You can see a summary of these in Figure 1. In the rest of this section, we will discuss each of these in-depth.
True
It may seem obvious but a good explanation should be true. This may be harder than expected when you consider what we are explaining. That is we are giving explanations for model predictions. The issue is that these predictions can be incorrect. For example, our model may be overfitted. This means predictions may reflect noise that has been modelled. An explanation for these predictions would not reflect the true underlying relationships in the data.
When giving explanations we need to consider how well our model represents reality. We can do this by evaluating the performance of the model. Such as by using cross-validation accuracy. Even with good overall performance, some predictions may be more uncertain than others. For example, logistic regression predictions around a 0.5 probability. In your explanation, we may want to mention this uncertainty.
Aimed at target audience
The way you phrase things is important. We rejected your loan because — “You are a crypto-currency trader”, “You work in a risky industry” or “Your income is too unstable”. These are all the same reasons but phrased differently. Some ways of phrasing things will better get your point across. Some may also find certain ways offensive. For example, the first explanation may be taken personally.
You should also refrain from any technical or business jargon. “Income has a positive parameter in our model and your income value is low. So, you were rejected as your income value has significantly increased your default risk.” This is a bad explanation as we have used technical jargon (i.e. “models” and “parameters”). We also have some business jargon (i.e. “default risk”).
Choosing the best way to phrase an explanation is really where the art comes in. It can be the most difficult part of explaining a prediction. It is also not something that is covered well in data science courses or university. Ultimately, this skill will come with experience and through developing relationships with customers.
Number of reasons
People only want the main reasons for an event. “Why is inflation so high?” — “Because oil prices have increased.” In reality, inflation can be caused by a combination of factors including wage growth, higher government spending or the exchange rate. However, these may not have had such a significant impact as rising energy costs.
When it comes to ML, people do not want to know how every model feature has impacted a prediction. Generally, explaining the contributions of 1 to 3 features is enough. The question is which contributions do you choose to explain? The next 4 characteristics can help you select which reasons are most appropriate.
Significant
We should select the reasons that are the most significant. In other words, we would only explain the factors that have had a large impact. For our inflation example, we would give the reasons that have caused the largest increases. These reasons will be more readily accepted than those that have had a smaller impact.
For ML, we want to explain the features that have contributed the most to a prediction. The way we determine these features will depend on what model has been used. For linear models, we can look at parameter weights and feature values. Later we see how SHAP values can be used for non-linear models.
General
A good reason could be one that explains many decisions. “Why did you reject my loan application?”- “Because you have a lot of existing debt.” Suppose this is a reason for rejecting many loan applications. This reason would be more readily accepted as its contribution is widely understood.
For ML, we can find general explanations by looking at some measure of feature importance. For example, permutation feature importance. With SHAP values we can look at the mean absolute SHAP. Whatever measure, a high value indicates that a feature has made significant contributions in general.
Abnormal
In other cases, a good reason could be one that is not common. These would be able to explain a significant contribution to a specific decision. However, in general, they would not be able to explain decisions. “Why did you reject my loan application?”- “Because you are a crypto-currency trader” These types of reasons may be accepted as they are more personal. That is the person can understand how it has impacted them specifically.
For ML, we can find abnormal reasons by looking at a combination of feature importance and contributions to the individual predictions. These would be the features that do not have a high feature importance. However, for a specific prediction, they have made a large contribution.
Contrasting
Often we will need to explain a decision in contrast to another decision. A customer may not ask “Why was my application rejected?” but “Why was my application rejected and theirs accepted?” It may be that both customers have high existing debt. This reason may have been accepted for the first question but not the second. In other words, we will need to give a reason that will distinguish the two decisions.
For ML, this means we need to base our explanation on features that have different values for the two customers. The feature will also need to differ in a way that would lead to different predictions. For example, we may find that the first customer has a higher income. The income feature is different but this would not be a good reason. That is higher income would not lead to an application being rejected.
These last four characteristics may seem to contradict each other. An abnormal reason cannot be general. An abnormal or general reason may not be the most significant. A contrasting reason can be neither significant, general or abnormal. However, we can use a combination of reasons in our explanation. The reasons you choose will depend on the question, person and what you believe will be most convincing.
Now let’s put some of these characteristics into practice. We will be trying to explain a model used to predict the amount charged for insurance (charges). You can see a snapshot of this dataset in Table 1. We base our model features on the first 6 columns. For example, children is the number of dependants.
We won’t go over the code but you can find the full project on GitHub. To summarise, we start by doing some feature engineering. We convert sex and smoker to binary features. region can take on 4 different values and so we create 4 dummy variables from this column. This leaves us with 9 model features in total. We use these to train an XGBoost model to predict charges. Once our model is trained we calculated the SHAP values for each prediction.
For each prediction, there will be a SHAP value for every feature. The SHAP value gives the contribution of that feature to the prediction. In other words, how much the feature increased or decreased the predicted charges. To understand which features are important we can look at Figure 2. Here we have taken the absolute mean of the SHAP values across all the predictions. A high mean indicates that the feature has, in general, made large contributions to the predictions.
We can use Figure 2 to help determine good general or abnormal reasons. For example, we can see that smoker has a high absolute mean. This tells us that, in general, this feature would provide a good reason for a prediction. The features with lower absolute means may provide good abnormal reasons. That is if, for a specific prediction, they have a large contribution.
Let’s try to explain our first prediction. You can see the SHAP waterfall plot for this prediction in Figure 3. Looking at the x-axis, we can see the base value is E[f(x)] = 12813. This is the average predicted charges across all the customers. The ending value is f(x) = 26726. This is the predicted charges for this specific customer. The SHAP values are all the values in between. They tell us how each feature has increased or decreased the prediction compared to the average prediction.
Looking at Figure 3 we can see that smoker is the most significant feature. That is it has increased the predicted charges by the largest amount. The feature values are given on the y-axis. For example, we can see that “1 = smoker” indicates that this customer smokes. So, if the customer asked “Why is my insurance charge so high?”, a good explanation could be “You are a smoker”. We also saw that, in Figure 2, this was a good general reason.
So smoking is both a significant and general reason. This may be enough to convince the person that that insurance charge is correct. If we wanted to be sure we could mention the second most significant feature. Looking at the y-axis we can see that the person is 62. Hence, we could follow up with a second reason, “and you are old.” (We might want to phrase this in a nicer way.)
For this second reason, we have guessed the relationship with the target variable. That is as age increases your charges increase. The relationship for other features may not be so obvious. This means to give good reasons we need some context for the feature values. To do this, we can use a scatter plot of the SHAP values. Looking at Figure 4, we can see our guess was correct. As age increases the SHAP values increase. In other words, the predicted charges increase.
In Figure 5, you can see the waterfall plot for a second prediction. To explain this one we may jump straight to the age of the customer. However, notice that children has also had a significant contribution. Remember, we saw in Figure 2 that this feature was not significant in general. In other words, the number of dependents may be a good abnormal reason. We may prefer to give this as the main reason instead.
By nature, SHAP values allow you to give contrasting explanations. However, this is only if we want to compare predictions to the average prediction. It is easy to answer questions like “Why is my charge higher than average?”. To answer questions like “Why is my charge higher than my sisters?” will take more work. There are other approaches to calculating SHAP values that can make this easier. For example, baseline SHAP will calculate the values w.r.t. a specific prediction.
We’ve focused on using SHAP values to explain individual predictions. They can also be used to give global explanations. That is explaining how the model works as a whole. We do this by aggregating SHAP values and creating different plots. We’ve already seen one — mean SHAP plot. We discuss more in the article below. We also discuss the Python code for calculating SHAP values and creating these plots.
How to explain your model in a human-friendly way
An important part of a data scientist’s role is to explain model predictions. Often, the person receiving the explanation will be non-technical. If you start talking about cost functions, hyperparameters or p-values you will be met with blank stares. We need to translate these technical concepts into layman’s terms. This process can be more challenging than building the model itself.
We will explore how you can give human-friendly explanations. We will do this by discussing some key characteristics of a good explanation. The focus will be on explaining individual predictions. In the end, we will apply some of this knowledge by explaining a model using SHAP values. We will see that SHAP is very useful when you want to give human-friendly explanations.
Before we dive into that, let’s discuss what you’re explaining and who you’re explaining it to. As a data scientist, you will likely need to communicate with a variety of people. This includes colleagues, regulators or customers. These people will all have various levels of technical expertise. So you will need to adjust the level of your explanation based on their expertise.
In some cases, you could be asked to explain the model as a whole. We refer to this as a global explanation. We want to understand what trends are being captured by the model in general. We may need to answer questions like “which features are the most important?” or “what relationship does feature X have with the target variable?”
In other cases, we will need to give local explanations. This is when we explain individual model predictions. In fact, we will usually have to explain a decision that has resulted from a model prediction. These explanations can also be answers to questions like “why did we reject this loan application?” or “why was I given this movie recommendation?”
When talking to a colleague or regulator you may need to give more technical explanations. In comparison, customers would expect simpler explanations. It is also unlikely that you would need to give a global explanation to a customer. This is because they would typically only be concerned with decisions that affect them personally. We will focus on this scenario. That is explaining individual predictions to a non-technical person.
When we talk about a good explanation we mean one that will be readily accepted. In other words, it should convince the audience that a decision was correct. To give such an explanation there are some key aspects you need to consider. You can see a summary of these in Figure 1. In the rest of this section, we will discuss each of these in-depth.
True
It may seem obvious but a good explanation should be true. This may be harder than expected when you consider what we are explaining. That is we are giving explanations for model predictions. The issue is that these predictions can be incorrect. For example, our model may be overfitted. This means predictions may reflect noise that has been modelled. An explanation for these predictions would not reflect the true underlying relationships in the data.
When giving explanations we need to consider how well our model represents reality. We can do this by evaluating the performance of the model. Such as by using cross-validation accuracy. Even with good overall performance, some predictions may be more uncertain than others. For example, logistic regression predictions around a 0.5 probability. In your explanation, we may want to mention this uncertainty.
Aimed at target audience
The way you phrase things is important. We rejected your loan because — “You are a crypto-currency trader”, “You work in a risky industry” or “Your income is too unstable”. These are all the same reasons but phrased differently. Some ways of phrasing things will better get your point across. Some may also find certain ways offensive. For example, the first explanation may be taken personally.
You should also refrain from any technical or business jargon. “Income has a positive parameter in our model and your income value is low. So, you were rejected as your income value has significantly increased your default risk.” This is a bad explanation as we have used technical jargon (i.e. “models” and “parameters”). We also have some business jargon (i.e. “default risk”).
Choosing the best way to phrase an explanation is really where the art comes in. It can be the most difficult part of explaining a prediction. It is also not something that is covered well in data science courses or university. Ultimately, this skill will come with experience and through developing relationships with customers.
Number of reasons
People only want the main reasons for an event. “Why is inflation so high?” — “Because oil prices have increased.” In reality, inflation can be caused by a combination of factors including wage growth, higher government spending or the exchange rate. However, these may not have had such a significant impact as rising energy costs.
When it comes to ML, people do not want to know how every model feature has impacted a prediction. Generally, explaining the contributions of 1 to 3 features is enough. The question is which contributions do you choose to explain? The next 4 characteristics can help you select which reasons are most appropriate.
Significant
We should select the reasons that are the most significant. In other words, we would only explain the factors that have had a large impact. For our inflation example, we would give the reasons that have caused the largest increases. These reasons will be more readily accepted than those that have had a smaller impact.
For ML, we want to explain the features that have contributed the most to a prediction. The way we determine these features will depend on what model has been used. For linear models, we can look at parameter weights and feature values. Later we see how SHAP values can be used for non-linear models.
General
A good reason could be one that explains many decisions. “Why did you reject my loan application?”- “Because you have a lot of existing debt.” Suppose this is a reason for rejecting many loan applications. This reason would be more readily accepted as its contribution is widely understood.
For ML, we can find general explanations by looking at some measure of feature importance. For example, permutation feature importance. With SHAP values we can look at the mean absolute SHAP. Whatever measure, a high value indicates that a feature has made significant contributions in general.
Abnormal
In other cases, a good reason could be one that is not common. These would be able to explain a significant contribution to a specific decision. However, in general, they would not be able to explain decisions. “Why did you reject my loan application?”- “Because you are a crypto-currency trader” These types of reasons may be accepted as they are more personal. That is the person can understand how it has impacted them specifically.
For ML, we can find abnormal reasons by looking at a combination of feature importance and contributions to the individual predictions. These would be the features that do not have a high feature importance. However, for a specific prediction, they have made a large contribution.
Contrasting
Often we will need to explain a decision in contrast to another decision. A customer may not ask “Why was my application rejected?” but “Why was my application rejected and theirs accepted?” It may be that both customers have high existing debt. This reason may have been accepted for the first question but not the second. In other words, we will need to give a reason that will distinguish the two decisions.
For ML, this means we need to base our explanation on features that have different values for the two customers. The feature will also need to differ in a way that would lead to different predictions. For example, we may find that the first customer has a higher income. The income feature is different but this would not be a good reason. That is higher income would not lead to an application being rejected.
These last four characteristics may seem to contradict each other. An abnormal reason cannot be general. An abnormal or general reason may not be the most significant. A contrasting reason can be neither significant, general or abnormal. However, we can use a combination of reasons in our explanation. The reasons you choose will depend on the question, person and what you believe will be most convincing.
Now let’s put some of these characteristics into practice. We will be trying to explain a model used to predict the amount charged for insurance (charges). You can see a snapshot of this dataset in Table 1. We base our model features on the first 6 columns. For example, children is the number of dependants.
We won’t go over the code but you can find the full project on GitHub. To summarise, we start by doing some feature engineering. We convert sex and smoker to binary features. region can take on 4 different values and so we create 4 dummy variables from this column. This leaves us with 9 model features in total. We use these to train an XGBoost model to predict charges. Once our model is trained we calculated the SHAP values for each prediction.
For each prediction, there will be a SHAP value for every feature. The SHAP value gives the contribution of that feature to the prediction. In other words, how much the feature increased or decreased the predicted charges. To understand which features are important we can look at Figure 2. Here we have taken the absolute mean of the SHAP values across all the predictions. A high mean indicates that the feature has, in general, made large contributions to the predictions.
We can use Figure 2 to help determine good general or abnormal reasons. For example, we can see that smoker has a high absolute mean. This tells us that, in general, this feature would provide a good reason for a prediction. The features with lower absolute means may provide good abnormal reasons. That is if, for a specific prediction, they have a large contribution.
Let’s try to explain our first prediction. You can see the SHAP waterfall plot for this prediction in Figure 3. Looking at the x-axis, we can see the base value is E[f(x)] = 12813. This is the average predicted charges across all the customers. The ending value is f(x) = 26726. This is the predicted charges for this specific customer. The SHAP values are all the values in between. They tell us how each feature has increased or decreased the prediction compared to the average prediction.
Looking at Figure 3 we can see that smoker is the most significant feature. That is it has increased the predicted charges by the largest amount. The feature values are given on the y-axis. For example, we can see that “1 = smoker” indicates that this customer smokes. So, if the customer asked “Why is my insurance charge so high?”, a good explanation could be “You are a smoker”. We also saw that, in Figure 2, this was a good general reason.
So smoking is both a significant and general reason. This may be enough to convince the person that that insurance charge is correct. If we wanted to be sure we could mention the second most significant feature. Looking at the y-axis we can see that the person is 62. Hence, we could follow up with a second reason, “and you are old.” (We might want to phrase this in a nicer way.)
For this second reason, we have guessed the relationship with the target variable. That is as age increases your charges increase. The relationship for other features may not be so obvious. This means to give good reasons we need some context for the feature values. To do this, we can use a scatter plot of the SHAP values. Looking at Figure 4, we can see our guess was correct. As age increases the SHAP values increase. In other words, the predicted charges increase.
In Figure 5, you can see the waterfall plot for a second prediction. To explain this one we may jump straight to the age of the customer. However, notice that children has also had a significant contribution. Remember, we saw in Figure 2 that this feature was not significant in general. In other words, the number of dependents may be a good abnormal reason. We may prefer to give this as the main reason instead.
By nature, SHAP values allow you to give contrasting explanations. However, this is only if we want to compare predictions to the average prediction. It is easy to answer questions like “Why is my charge higher than average?”. To answer questions like “Why is my charge higher than my sisters?” will take more work. There are other approaches to calculating SHAP values that can make this easier. For example, baseline SHAP will calculate the values w.r.t. a specific prediction.
We’ve focused on using SHAP values to explain individual predictions. They can also be used to give global explanations. That is explaining how the model works as a whole. We do this by aggregating SHAP values and creating different plots. We’ve already seen one — mean SHAP plot. We discuss more in the article below. We also discuss the Python code for calculating SHAP values and creating these plots.
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.