What we can Learn from Black-box Models | by Conor O’Sullivan | Jul, 2022
Data exploration and knowledge generation using non-linear models
Black-box/non-linear models can automatically model complex relationships in data. Capturing these relationships is what increases their accuracy compared to linear models. However, accurate predictions are only one of the benefits. We can analyse the black-box models to learn how they make those predictions. This can reveal underlying relationships in our dataset. In some cases, these can be completely new to us. In this way, machine learning becomes a tool for data exploration and knowledge generation.
We will discuss what you can learn using black-box models. This includes important relationships in data and the nature of those relationships. At the same time, we discuss how we find and visualise these relationships. We then move on to discussing how you can use this knowledge. That is, it can be used as a sense check on the actual model or to inform feature engineering for non-linear models. The knowledge can also help when making decisions that go beyond models.
Before we dive into that, let’s discuss why we can learn from these models. Non-linear models can model non-linear relationships and interactions in our data. For example, take the scatter plot of second-hand car prices in Figure 1. Car price does not always decrease with age. For classic cars (classic = 1), it increases. We say there is an interaction in our data. The relationship between price and age depends on a third feature- classic.
Non-linear models can automatically model relationships like these. Let’s take the simple decision tree in Figure 2. The cars are first split into normal (0) and classic (1). For normal cars, the price is higher if the car is less than 15 years old. For classic cars, it is lower. This doesn’t perfectly capture the relationship we saw in Figure 1. To do that we could add more layers to the decision trees. We could also use more complicated algorithms like random forests, XGBoost or neural networks.
When we talk about data exploration/ knowledge generation we will usually rely on these more complicated algorithms. They are sometimes referred to as explainable models or black-box models. They are able to capture complicated relationships in data. At the same time, we do not need to “tell” them that those relationships exist. Like with the decision tree, we can give them the classic and age features only. The models will automatically make use of the interaction between the features when making predictions.
In comparison, linear models (i.e. linear and logistic regression) are less complicated. To capture non-linear relationships with these models we need to do feature engineering. Using new features, we can represent non-linear relationships as linear ones. To understand what feature to create, we will need to explore our data. We’ll likely also need domain knowledge in the relevant field.
In a sense, black-box models do this feature engineering within the model’s structure. They are incredibly good at finding patterns in data. The issue is that the inner workings of the models are too complex. We cannot understand how they make predictions by looking at the model parameters/ structure alone. We need to use additional techniques to understand how they work.
This is where the field of interpretable machine learning (IML) comes in. Amongst other things, it aims to develop methods we can use to understand how black-box models make predictions. We discuss some of these techniques and what they can teach us about models/data.
Provide a Benchmark
Firstly, a valuable piece of information that black-box models can provide is a benchmark. By capturing non-linear relationships and interactions these models can improve the accuracy of predictions. To determine performance, we don’t actually need IML methods. We can rely on metrics like accuracy, precision or recall or visualisations like ROC curves.
Using black-box models can tell us how well we are able to predict the target variable. We can compare these results to the performance of less complicated linear models. Suppose, we find the black-box models are significantly more accurate. This tells us that the linear models are missing important relationships in our data. At this point, we can use IML method to understand what those relationships are.
Important relationships
To start we can understand which features in our data are important. These are the features that can be used to predict the target variable. Most non-linear models are able to handle a large number of features (over 50). It also doesn’t matter if those features are highly correlated. This means, that by using non-linear models, we can understand the importance of a large set of features. In comparison, models like linear regression require a smaller set (8–10) of uncorrelated features.
One measure we can use is permutation feature importance. In Figure 3, you can see the scores from an xgboost model used to predict the age of an abalone. We can see the length of an abalone shell was important when predicting age. The next three features are based on the weight of the abalone. Although these features are highly correlated, they have not had a negative impact on the model. We can understand that all these features could be used to predict age.
If we use correlation to identify important relationships we may miss some of these features. This is because correlation only gives a measure of linear association. In other words, this measure can only highlight features that have a significant linear relationship. In comparison, feature importance tells us how important a feature is to a model’s predictions. A model’s ability to predict could come from interactions and non-linear relationships. These will be reflected in the feature importance score.
SHAP values can provide an alternative measure. We can use these to find the absolute mean SHAP value for each feature. Similar to feature importance, this can tell which features have made large contributions to predictions. The issue with both of these measures is they do not tell us the nature of the relationships captured by the model.
Non-linear relationships
A feature can be used to make predictions if it has any kind of relationship with the target variable. It could be linear or, for example, one of the non-linear relationships seen in Figure 4. To find and visualise non-linear relationships captured by a black-box model, we can use other IML methods.
Examples of visualisations are partial dependence plots (PDP) and accumulated local effects (ALE) plots. Both of these methods show the general relationship between a feature and the model’s predictions. PDPs are easy to understand but they can be negatively impacted by feature dependencies. ALE Plots don’t have this problem but they are more complicated and harder to explain.
Another approach is to use SHAP values. We mentioned these above. For a given prediction, they give the contribution of each feature to that prediction. We can plot the SHAP values vs the feature values. This is known as a dependencies plot. It can show us the relationship between a feature’s value and its contributions to predictions.
We discuss these and more methods in the articles below. The first goes into more depth on how to find and visualise non-linear relationships. The second focuses on SHAP values and the Python code used to apply them. We also go over different aggregations of these values. These include the mean SHAP and dependencies plots mentioned above.
Interactions
Interactions are actually considered to be a type of non-linear relationship. They are non-linear w.r.t. to one feature. The relationship of that feature will depend on the value of a third feature. In fact, interactions can be even more complicated. The relationship of a feature could depend on the values of two or more features. For most applications, we are only concerned with those between two features.
One method of visualising interactions is ICE Plots. You can see an example in Figure 5. Here each observation in our dataset is represented by a line. This dataset contains details on the sale of second-hand cars (e.g. the price, age and type of car). For each observation, the line shows how the predicted price (partial yhat) changes with car age. The bold yellow is an example of a PDP we discussed above. It gives the average of all the individual prediction lines.
We have coloured the above lines by the car type. Blue for classic cars and red for normal cars. You can see that the relationship for classic cars deviates from the average trend. Intuitively, it makes sense that the price of a classic would increase with age. The ICE Plot shows us that the relationship between car age and price depends on the type of car. In other words, there is an interaction between car age and car type.
Another approach is to use SHAP interaction values. These are an extension of SHAP values where both main and interaction effects are considered. We can use these to visualise interactions. We can also calculate the mean absolute interaction effect to highlight important interactions. This is similar to a feature importance score but for interactions. Another metric used to highlight interactions is the Friedman’s h-statistic.
Again, we discuss all of these methods in more depth in the articles below. The first goes into more depth on how to find and visualise interactions. The second focuses on using SHAP to analyse interactions. We also discuss the Python code used to apply this method.
All these IML methods can help us understand how a model makes predictions. A direct benefit is they can help us sense-check our models. We can compare the relationships captured by the model to those we expected using domain knowledge. These methods also allow us to explain our models to customers or colleagues.
In this section, we will concentrate on the indirect benefits. By understanding the model we learn about the underlying relationships in the data. We can use this knowledge to improve linear models or to enable decision-making that goes beyond the model.
Improve linear models
We mentioned that black-box models can provide a benchmark. They may outperform linear models but this does not mean the end of these models. This is because we can use the knowledge we gain to inform our feature engineering. We can create new features that will improve the performance of our linear models.
For example, suppose we wanted to model the price of a second-hand car using linear regression. Using PDPs, we discovered that the price has a quadratic relationship with the car age. Similarly, using ICE Plots, we discovered an interaction between car age and car type. We can capture these non-linear relationships by adding new features to our dataset.
Specifically, in the equation below we have added age² and the interaction term age*classic. Where classic = 1 for classic cars and 0 otherwise.
Y = β₀ + β₁(age) + β₂(age²) + β₃(age*classic) + β₄(classic)
Y = β₀ + β₁(age) + β₂(age_sqrd) + β₃(age_classic_int) + β₄(classic)
In the first equation, Y was modelled by a non-linear equation of two features — age and classic. Now, it is modelled by a linear equation of four features. Feature engineering can even help simple non-linear models like decision trees. Ultimately, it allows us to capture complex relationships with simple models. We demonstrate this process more practically in the article below.
The question is why would we want to do this? A non-linear model is just as accurate. So why not use them and avoid the hassle of additional feature engineering? It is because there are many benefits to having a simple model. For one, in many industries, there are regulations on the types of models you can use. That is you may be required to use linear models in industries like banking and insurance.
Even if there is no regulation you may still prefer linear models. Black-box models are complicated. Even with IML methods, we may not be 100% sure about how they work. This can lead to surprises when they are put into production. In comparison, regression and decision trees are intrinsically interpretable. We can understand how they work by looking directly at their parameters. This increases their certainty about how they will behave in the wild.
Enable decision making
Models are used to automate decisions. With IML their impact is no longer restricted to only those decisions. The knowledge we gain can inform decisions in the wider organisation. Relationships that were previously not known can provide great value. For example, the interaction we explored tells us that classic cars will increase in price as they age. This may inform a strategy of investing in classic cars.
That being said we should avoid over-complicated solutions. Using data to inform decisions is not new to data scientists. We have many techniques and tools we can use to help explore data. Most are them are much simpler than using black-box models. the downside is they do require some direction. We need an idea of what we want to find in the data. With IML methods, black-box models can find new important patterns for us.
Data exploration and knowledge generation using non-linear models
Black-box/non-linear models can automatically model complex relationships in data. Capturing these relationships is what increases their accuracy compared to linear models. However, accurate predictions are only one of the benefits. We can analyse the black-box models to learn how they make those predictions. This can reveal underlying relationships in our dataset. In some cases, these can be completely new to us. In this way, machine learning becomes a tool for data exploration and knowledge generation.
We will discuss what you can learn using black-box models. This includes important relationships in data and the nature of those relationships. At the same time, we discuss how we find and visualise these relationships. We then move on to discussing how you can use this knowledge. That is, it can be used as a sense check on the actual model or to inform feature engineering for non-linear models. The knowledge can also help when making decisions that go beyond models.
Before we dive into that, let’s discuss why we can learn from these models. Non-linear models can model non-linear relationships and interactions in our data. For example, take the scatter plot of second-hand car prices in Figure 1. Car price does not always decrease with age. For classic cars (classic = 1), it increases. We say there is an interaction in our data. The relationship between price and age depends on a third feature- classic.
Non-linear models can automatically model relationships like these. Let’s take the simple decision tree in Figure 2. The cars are first split into normal (0) and classic (1). For normal cars, the price is higher if the car is less than 15 years old. For classic cars, it is lower. This doesn’t perfectly capture the relationship we saw in Figure 1. To do that we could add more layers to the decision trees. We could also use more complicated algorithms like random forests, XGBoost or neural networks.
When we talk about data exploration/ knowledge generation we will usually rely on these more complicated algorithms. They are sometimes referred to as explainable models or black-box models. They are able to capture complicated relationships in data. At the same time, we do not need to “tell” them that those relationships exist. Like with the decision tree, we can give them the classic and age features only. The models will automatically make use of the interaction between the features when making predictions.
In comparison, linear models (i.e. linear and logistic regression) are less complicated. To capture non-linear relationships with these models we need to do feature engineering. Using new features, we can represent non-linear relationships as linear ones. To understand what feature to create, we will need to explore our data. We’ll likely also need domain knowledge in the relevant field.
In a sense, black-box models do this feature engineering within the model’s structure. They are incredibly good at finding patterns in data. The issue is that the inner workings of the models are too complex. We cannot understand how they make predictions by looking at the model parameters/ structure alone. We need to use additional techniques to understand how they work.
This is where the field of interpretable machine learning (IML) comes in. Amongst other things, it aims to develop methods we can use to understand how black-box models make predictions. We discuss some of these techniques and what they can teach us about models/data.
Provide a Benchmark
Firstly, a valuable piece of information that black-box models can provide is a benchmark. By capturing non-linear relationships and interactions these models can improve the accuracy of predictions. To determine performance, we don’t actually need IML methods. We can rely on metrics like accuracy, precision or recall or visualisations like ROC curves.
Using black-box models can tell us how well we are able to predict the target variable. We can compare these results to the performance of less complicated linear models. Suppose, we find the black-box models are significantly more accurate. This tells us that the linear models are missing important relationships in our data. At this point, we can use IML method to understand what those relationships are.
Important relationships
To start we can understand which features in our data are important. These are the features that can be used to predict the target variable. Most non-linear models are able to handle a large number of features (over 50). It also doesn’t matter if those features are highly correlated. This means, that by using non-linear models, we can understand the importance of a large set of features. In comparison, models like linear regression require a smaller set (8–10) of uncorrelated features.
One measure we can use is permutation feature importance. In Figure 3, you can see the scores from an xgboost model used to predict the age of an abalone. We can see the length of an abalone shell was important when predicting age. The next three features are based on the weight of the abalone. Although these features are highly correlated, they have not had a negative impact on the model. We can understand that all these features could be used to predict age.
If we use correlation to identify important relationships we may miss some of these features. This is because correlation only gives a measure of linear association. In other words, this measure can only highlight features that have a significant linear relationship. In comparison, feature importance tells us how important a feature is to a model’s predictions. A model’s ability to predict could come from interactions and non-linear relationships. These will be reflected in the feature importance score.
SHAP values can provide an alternative measure. We can use these to find the absolute mean SHAP value for each feature. Similar to feature importance, this can tell which features have made large contributions to predictions. The issue with both of these measures is they do not tell us the nature of the relationships captured by the model.
Non-linear relationships
A feature can be used to make predictions if it has any kind of relationship with the target variable. It could be linear or, for example, one of the non-linear relationships seen in Figure 4. To find and visualise non-linear relationships captured by a black-box model, we can use other IML methods.
Examples of visualisations are partial dependence plots (PDP) and accumulated local effects (ALE) plots. Both of these methods show the general relationship between a feature and the model’s predictions. PDPs are easy to understand but they can be negatively impacted by feature dependencies. ALE Plots don’t have this problem but they are more complicated and harder to explain.
Another approach is to use SHAP values. We mentioned these above. For a given prediction, they give the contribution of each feature to that prediction. We can plot the SHAP values vs the feature values. This is known as a dependencies plot. It can show us the relationship between a feature’s value and its contributions to predictions.
We discuss these and more methods in the articles below. The first goes into more depth on how to find and visualise non-linear relationships. The second focuses on SHAP values and the Python code used to apply them. We also go over different aggregations of these values. These include the mean SHAP and dependencies plots mentioned above.
Interactions
Interactions are actually considered to be a type of non-linear relationship. They are non-linear w.r.t. to one feature. The relationship of that feature will depend on the value of a third feature. In fact, interactions can be even more complicated. The relationship of a feature could depend on the values of two or more features. For most applications, we are only concerned with those between two features.
One method of visualising interactions is ICE Plots. You can see an example in Figure 5. Here each observation in our dataset is represented by a line. This dataset contains details on the sale of second-hand cars (e.g. the price, age and type of car). For each observation, the line shows how the predicted price (partial yhat) changes with car age. The bold yellow is an example of a PDP we discussed above. It gives the average of all the individual prediction lines.
We have coloured the above lines by the car type. Blue for classic cars and red for normal cars. You can see that the relationship for classic cars deviates from the average trend. Intuitively, it makes sense that the price of a classic would increase with age. The ICE Plot shows us that the relationship between car age and price depends on the type of car. In other words, there is an interaction between car age and car type.
Another approach is to use SHAP interaction values. These are an extension of SHAP values where both main and interaction effects are considered. We can use these to visualise interactions. We can also calculate the mean absolute interaction effect to highlight important interactions. This is similar to a feature importance score but for interactions. Another metric used to highlight interactions is the Friedman’s h-statistic.
Again, we discuss all of these methods in more depth in the articles below. The first goes into more depth on how to find and visualise interactions. The second focuses on using SHAP to analyse interactions. We also discuss the Python code used to apply this method.
All these IML methods can help us understand how a model makes predictions. A direct benefit is they can help us sense-check our models. We can compare the relationships captured by the model to those we expected using domain knowledge. These methods also allow us to explain our models to customers or colleagues.
In this section, we will concentrate on the indirect benefits. By understanding the model we learn about the underlying relationships in the data. We can use this knowledge to improve linear models or to enable decision-making that goes beyond the model.
Improve linear models
We mentioned that black-box models can provide a benchmark. They may outperform linear models but this does not mean the end of these models. This is because we can use the knowledge we gain to inform our feature engineering. We can create new features that will improve the performance of our linear models.
For example, suppose we wanted to model the price of a second-hand car using linear regression. Using PDPs, we discovered that the price has a quadratic relationship with the car age. Similarly, using ICE Plots, we discovered an interaction between car age and car type. We can capture these non-linear relationships by adding new features to our dataset.
Specifically, in the equation below we have added age² and the interaction term age*classic. Where classic = 1 for classic cars and 0 otherwise.
Y = β₀ + β₁(age) + β₂(age²) + β₃(age*classic) + β₄(classic)
Y = β₀ + β₁(age) + β₂(age_sqrd) + β₃(age_classic_int) + β₄(classic)
In the first equation, Y was modelled by a non-linear equation of two features — age and classic. Now, it is modelled by a linear equation of four features. Feature engineering can even help simple non-linear models like decision trees. Ultimately, it allows us to capture complex relationships with simple models. We demonstrate this process more practically in the article below.
The question is why would we want to do this? A non-linear model is just as accurate. So why not use them and avoid the hassle of additional feature engineering? It is because there are many benefits to having a simple model. For one, in many industries, there are regulations on the types of models you can use. That is you may be required to use linear models in industries like banking and insurance.
Even if there is no regulation you may still prefer linear models. Black-box models are complicated. Even with IML methods, we may not be 100% sure about how they work. This can lead to surprises when they are put into production. In comparison, regression and decision trees are intrinsically interpretable. We can understand how they work by looking directly at their parameters. This increases their certainty about how they will behave in the wild.
Enable decision making
Models are used to automate decisions. With IML their impact is no longer restricted to only those decisions. The knowledge we gain can inform decisions in the wider organisation. Relationships that were previously not known can provide great value. For example, the interaction we explored tells us that classic cars will increase in price as they age. This may inform a strategy of investing in classic cars.
That being said we should avoid over-complicated solutions. Using data to inform decisions is not new to data scientists. We have many techniques and tools we can use to help explore data. Most are them are much simpler than using black-box models. the downside is they do require some direction. We need an idea of what we want to find in the data. With IML methods, black-box models can find new important patterns for us.
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.