Causal Inference Using Synthetic Control | by diksha tiwari | Nov, 2022
Exploring one of the latest quasi-experimentation techniques
It is widely accepted that A/B testing is the gold standard for causal inference. Also known as Randomized Controlled Tests (RCTs), these tests involve splitting the subjects randomly into treatment and control units. This ensures that any difference between the units is due to the applied treatment. A/B testing has been widely adopted by businesses to test new products, features, and marketing strategies. This helps them capture customer reactions, issues with the product, etc. early on in a product or strategy cycle. However, there are numerous situations where randomly splitting the subjects into treatment and control groups may not be the best solution. An example could be social media testing, where network effects may lead to contamination between test and control. Similarly, in some cases, it may raise ethical concerns (for example, in medical trials) or may be too expensive or even rendered infeasible due to technical limitations. It is in these situations that we use quasi-experimental techniques such as Difference in Differences Analysis (DID), matched pair testing, regression, etc. Synthetic control method, which is the focus of this article, is one such technique. This article explores:
1) the details of the synthetic control method,
2) its advantages and disadvantages, and
3) the data requirements for the technique
Synthetic Control Method (SCM):
What is a synthetic control method?
Synthetic control methods were originally proposed in Abadie and Gardeazabal (2003) with the aim of estimating the effects of aggregate interventions on some aggregate outcome of interest [1]. Here, aggregate interventions are the interventions that are implemented at an aggregate level and affect a small number of large units (such as cities, regions, or countries). It is based on the idea that when the observation is made at an aggregate entity level, a combination of unaffected units may provide a more appropriate comparison than any single unaffected unit. To put it simply, it compares the treatment group to a weighted combination of control groups. SCM has two major advantages over traditional quasi-experimental techniques:
1) It can account for the effect of confounders changing over time, whereas DID’s parallel trend assumption implies that, without the intervention, outcomes for the treated and control groups would have followed parallel trajectories over time [1].
2) This methodology formalizes the selection of the comparison units using a data-driven procedure [2].
Model details:
Let us assume that there are J units with j=1 being the treatment and j=2, …, j=J being the control units. Let Y be the outcome variable, Y_(1t)^N be the value of outcome variable that would have been observed for the treatment unit in the absence of intervention at time t. Let T_0 be the time of intervention, Y_(1t) be the value of the outcome variable post the intervention, and Y_(jt) be the value of the outcome variable for control unit j at time t. Using w_j as the weight associated with the control units, value of Y_(1t)^N can be represented as below:
If is the impact of intervention on treatment, then
Here, Y_(1t) can be obtained by observing Y post the intervention whereas Y_(1t)^N is obtained from Eq. (1). The question remains: how do we obtain the weights for the above equation? Abadie, Diamond, and Hainmueller (2010) propose calculating the weights in a manner similar to
Here W is a (J-1)x1 matrix of weights w_j, X_(t,pre) is the vector of pre-intervention characteristics for the exposed region, and X_(c,pre) is the vector of the same pre-intervention characteristics for the controls.
The pre-intervention characteristics, a.k.a. covariates can be any variables that appropriately represent the treatment. For instance, in Abadie, Diamond, and Hainmueller (2010), while estimating the impact of Proposition 99 on California, the covariates that were used were the average retail price of cigarettes in the pre-intervention period, the mean of per capita state personal income (logged) in the pre-intervention period, the percentage of the population aged 15–24 in the pre-intervention period, and the mean of per capita beer consumption in the pre-intervention period. These variables have been augmented by three years of lagged smoking consumption (which is also the outcome variable). One may use any number of years of lag data to model the treatment unit. [4]
The formula to calculate the weights for the model although quite similar to linear regression has subtle differences. The model uses following constraints which makes it different from the classical linear regression model:
The last two constraints safeguard the method against extrapolation. Because a synthetic control is a weighted average of the available control units, this method makes explicit: (1) the relative contribution of each control unit to the counterfactual of interest; and (2) the similarities (or lack thereof) between the unit affected by the event or intervention of interest and the synthetic control, in terms of pre-intervention outcomes and other predictors of post-intervention outcomes. Relative to traditional regression methods, transparency and a safeguard against extrapolation are the two attractive features of the method [4].
Implementation Example:
For this exercise, I have used publicly available data the details of which are described in [6]. The code from the article Understanding Synthetic Control Methods has been used for this example.
In this example, we will try to estimate the impact of Proposition 99 on annual per capita cigarette consumption at the state level which is measured as the per capita cigarette sale in our data-set. Thus, for this example our outcome variable of interest is “annual per capita cigarette sale”. The sample period for our example begins in 1970 and ends in 2000. California introduced Proposition 99 in 1989. Let us start by looking at the contextual requirements of this method [1]:
- Small effects with high volatility will be difficult to measure with this method.
- Availability of comparison group i.e. not all units adopt interventions similar to the treatment group. For this example, the states that introduced formal state-wide tobacco control programs or increased tax on cigarettes by more than 50 cents during the time period of our study were excluded
- No spill-over effect on the control units i.e. implementing intervention on treatment does not impact the outcome variable of interest in control units. This example assumes that there is no spillover effect between treatment and control units.
- Pre-intervention, differences in characteristics of synthetic control and affected unit is small i.e.
Given we have taken care of the contextual requirements let us look at the data:
As we see in the above image our data has three columns: State, year and cigsale which is our outcome variable of interest aka “annual per capita cigarette sale”. Since we will be using lagged data as covariates for our exercise; we need to transform it into panel data i.e. each row will represent the year and each column will represent the state.
Next is selecting the treatment state, which in our case is California and treatment year is 1989, the first year when Proposition 99 was introduced in California. Before moving onto constructing a synthetic California for our study let us compare the cig sale in California with the mean of cig sale in the rest of the control states in this study.
Looking at the above figure, it does seem like cig sales in California post 1989 declined more rapidly than the rest of control states. Next step is constructing synthetic California. We will first define our regression function with the below constraints:
The function to predict synthetic California:
Let us now construct our synthetic California and look at the weights of the control states:
The coefficients/weights obtained from this exercise are almost similar to the ones mentioned in [4]. The slight variances are due to the difference in covariates that were used in [4] and in this example. It is now time to compare the synthetic California with observed California.
Looking at the above plot, it looks like the effect of Proposition 99 on cig. sale is negative, however let’s visualize the difference between observed and synthetic California:
The difference plot is able to bring out the negative difference between synthetic and observed California especially after 1988 which is our treatment year. Although we now we could observe the negative effect of Propsition 99 on cig sales in California but the important question which remains is how do we determine that this effect is statistically significant. This bring us to the last section of the article which is Inference.
Inference:
Now that we have constructed our synthetic California and identified the difference between synthetic and observed California, how do we estimate the statistical significance of our study? To simply put it, how do we determine that the impact observed for California is not a chance occurrence. We will use the Permutation Test also described as a placebo study in [4]. We will apply synthetic control methods to the states that did not implement Proposition 99 during the time period of our study. The idea is that by applying synthetic control to other states if we do not observe an impact as big as California, then the impact of Proposition 99 on California is significant.
- Creating synthetic control for all the states and plotting the difference between synthetic and observed states:
We could observe in the above graph that for some states we did not get a very good fit. In [4], it is suggested that we remove states with MSE > 2*MSE of treatment state.
2. Let’s remove the states with MSE > 2*MSE of treatment state and look at the estimated difference between synthetic control and states.
After excluding the extreme states it looks like the effect of Proposition 99 on California is rare. In [4], the ratio between MSE Pre and MSE Post was used for the permutation test.
3. Let us calculate that ratio and estimate the p-value for the test:
The p-value for our test is 0.0256 i.e. if one were to assign the intervention at random in this data, the probability of obtaining a MSE post/ MSE pre as large as California is ~0.026.
References:
[1] Abadie, A. (2021). Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), 391–425.
[2] Abadie, A., Diamond, A., & Hainmueller, J. (2015). Comparative politics and the synthetic control method. American Journal of Political Science, 59(2), 495–510.
[3] Abadie, A., & Gardeazabal, J. (2003). The economic costs of conflict: A case study of the Basque Country. American economic review, 93(1), 113–132.
[4] Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American statistical Association, 105(490), 493–505.
[5] Doudchenko, N., & Imbens, G. W. (2016). Balancing, regression, difference-in-differences and synthetic control methods: A synthesis (No. w22791). National Bureau of Economic Research.
[6] Data source — Per-capita cigarette consumption (in packs). Source: Orzechowski and Walker (2005), Access & Use Information — Public
Exploring one of the latest quasi-experimentation techniques
It is widely accepted that A/B testing is the gold standard for causal inference. Also known as Randomized Controlled Tests (RCTs), these tests involve splitting the subjects randomly into treatment and control units. This ensures that any difference between the units is due to the applied treatment. A/B testing has been widely adopted by businesses to test new products, features, and marketing strategies. This helps them capture customer reactions, issues with the product, etc. early on in a product or strategy cycle. However, there are numerous situations where randomly splitting the subjects into treatment and control groups may not be the best solution. An example could be social media testing, where network effects may lead to contamination between test and control. Similarly, in some cases, it may raise ethical concerns (for example, in medical trials) or may be too expensive or even rendered infeasible due to technical limitations. It is in these situations that we use quasi-experimental techniques such as Difference in Differences Analysis (DID), matched pair testing, regression, etc. Synthetic control method, which is the focus of this article, is one such technique. This article explores:
1) the details of the synthetic control method,
2) its advantages and disadvantages, and
3) the data requirements for the technique
Synthetic Control Method (SCM):
What is a synthetic control method?
Synthetic control methods were originally proposed in Abadie and Gardeazabal (2003) with the aim of estimating the effects of aggregate interventions on some aggregate outcome of interest [1]. Here, aggregate interventions are the interventions that are implemented at an aggregate level and affect a small number of large units (such as cities, regions, or countries). It is based on the idea that when the observation is made at an aggregate entity level, a combination of unaffected units may provide a more appropriate comparison than any single unaffected unit. To put it simply, it compares the treatment group to a weighted combination of control groups. SCM has two major advantages over traditional quasi-experimental techniques:
1) It can account for the effect of confounders changing over time, whereas DID’s parallel trend assumption implies that, without the intervention, outcomes for the treated and control groups would have followed parallel trajectories over time [1].
2) This methodology formalizes the selection of the comparison units using a data-driven procedure [2].
Model details:
Let us assume that there are J units with j=1 being the treatment and j=2, …, j=J being the control units. Let Y be the outcome variable, Y_(1t)^N be the value of outcome variable that would have been observed for the treatment unit in the absence of intervention at time t. Let T_0 be the time of intervention, Y_(1t) be the value of the outcome variable post the intervention, and Y_(jt) be the value of the outcome variable for control unit j at time t. Using w_j as the weight associated with the control units, value of Y_(1t)^N can be represented as below:
If is the impact of intervention on treatment, then
Here, Y_(1t) can be obtained by observing Y post the intervention whereas Y_(1t)^N is obtained from Eq. (1). The question remains: how do we obtain the weights for the above equation? Abadie, Diamond, and Hainmueller (2010) propose calculating the weights in a manner similar to
Here W is a (J-1)x1 matrix of weights w_j, X_(t,pre) is the vector of pre-intervention characteristics for the exposed region, and X_(c,pre) is the vector of the same pre-intervention characteristics for the controls.
The pre-intervention characteristics, a.k.a. covariates can be any variables that appropriately represent the treatment. For instance, in Abadie, Diamond, and Hainmueller (2010), while estimating the impact of Proposition 99 on California, the covariates that were used were the average retail price of cigarettes in the pre-intervention period, the mean of per capita state personal income (logged) in the pre-intervention period, the percentage of the population aged 15–24 in the pre-intervention period, and the mean of per capita beer consumption in the pre-intervention period. These variables have been augmented by three years of lagged smoking consumption (which is also the outcome variable). One may use any number of years of lag data to model the treatment unit. [4]
The formula to calculate the weights for the model although quite similar to linear regression has subtle differences. The model uses following constraints which makes it different from the classical linear regression model:
The last two constraints safeguard the method against extrapolation. Because a synthetic control is a weighted average of the available control units, this method makes explicit: (1) the relative contribution of each control unit to the counterfactual of interest; and (2) the similarities (or lack thereof) between the unit affected by the event or intervention of interest and the synthetic control, in terms of pre-intervention outcomes and other predictors of post-intervention outcomes. Relative to traditional regression methods, transparency and a safeguard against extrapolation are the two attractive features of the method [4].
Implementation Example:
For this exercise, I have used publicly available data the details of which are described in [6]. The code from the article Understanding Synthetic Control Methods has been used for this example.
In this example, we will try to estimate the impact of Proposition 99 on annual per capita cigarette consumption at the state level which is measured as the per capita cigarette sale in our data-set. Thus, for this example our outcome variable of interest is “annual per capita cigarette sale”. The sample period for our example begins in 1970 and ends in 2000. California introduced Proposition 99 in 1989. Let us start by looking at the contextual requirements of this method [1]:
- Small effects with high volatility will be difficult to measure with this method.
- Availability of comparison group i.e. not all units adopt interventions similar to the treatment group. For this example, the states that introduced formal state-wide tobacco control programs or increased tax on cigarettes by more than 50 cents during the time period of our study were excluded
- No spill-over effect on the control units i.e. implementing intervention on treatment does not impact the outcome variable of interest in control units. This example assumes that there is no spillover effect between treatment and control units.
- Pre-intervention, differences in characteristics of synthetic control and affected unit is small i.e.
Given we have taken care of the contextual requirements let us look at the data:
As we see in the above image our data has three columns: State, year and cigsale which is our outcome variable of interest aka “annual per capita cigarette sale”. Since we will be using lagged data as covariates for our exercise; we need to transform it into panel data i.e. each row will represent the year and each column will represent the state.
Next is selecting the treatment state, which in our case is California and treatment year is 1989, the first year when Proposition 99 was introduced in California. Before moving onto constructing a synthetic California for our study let us compare the cig sale in California with the mean of cig sale in the rest of the control states in this study.
Looking at the above figure, it does seem like cig sales in California post 1989 declined more rapidly than the rest of control states. Next step is constructing synthetic California. We will first define our regression function with the below constraints:
The function to predict synthetic California:
Let us now construct our synthetic California and look at the weights of the control states:
The coefficients/weights obtained from this exercise are almost similar to the ones mentioned in [4]. The slight variances are due to the difference in covariates that were used in [4] and in this example. It is now time to compare the synthetic California with observed California.
Looking at the above plot, it looks like the effect of Proposition 99 on cig. sale is negative, however let’s visualize the difference between observed and synthetic California:
The difference plot is able to bring out the negative difference between synthetic and observed California especially after 1988 which is our treatment year. Although we now we could observe the negative effect of Propsition 99 on cig sales in California but the important question which remains is how do we determine that this effect is statistically significant. This bring us to the last section of the article which is Inference.
Inference:
Now that we have constructed our synthetic California and identified the difference between synthetic and observed California, how do we estimate the statistical significance of our study? To simply put it, how do we determine that the impact observed for California is not a chance occurrence. We will use the Permutation Test also described as a placebo study in [4]. We will apply synthetic control methods to the states that did not implement Proposition 99 during the time period of our study. The idea is that by applying synthetic control to other states if we do not observe an impact as big as California, then the impact of Proposition 99 on California is significant.
- Creating synthetic control for all the states and plotting the difference between synthetic and observed states:
We could observe in the above graph that for some states we did not get a very good fit. In [4], it is suggested that we remove states with MSE > 2*MSE of treatment state.
2. Let’s remove the states with MSE > 2*MSE of treatment state and look at the estimated difference between synthetic control and states.
After excluding the extreme states it looks like the effect of Proposition 99 on California is rare. In [4], the ratio between MSE Pre and MSE Post was used for the permutation test.
3. Let us calculate that ratio and estimate the p-value for the test:
The p-value for our test is 0.0256 i.e. if one were to assign the intervention at random in this data, the probability of obtaining a MSE post/ MSE pre as large as California is ~0.026.
References:
[1] Abadie, A. (2021). Using synthetic controls: Feasibility, data requirements, and methodological aspects. Journal of Economic Literature, 59(2), 391–425.
[2] Abadie, A., Diamond, A., & Hainmueller, J. (2015). Comparative politics and the synthetic control method. American Journal of Political Science, 59(2), 495–510.
[3] Abadie, A., & Gardeazabal, J. (2003). The economic costs of conflict: A case study of the Basque Country. American economic review, 93(1), 113–132.
[4] Abadie, A., Diamond, A., & Hainmueller, J. (2010). Synthetic control methods for comparative case studies: Estimating the effect of California’s tobacco control program. Journal of the American statistical Association, 105(490), 493–505.
[5] Doudchenko, N., & Imbens, G. W. (2016). Balancing, regression, difference-in-differences and synthetic control methods: A synthesis (No. w22791). National Bureau of Economic Research.
[6] Data source — Per-capita cigarette consumption (in packs). Source: Orzechowski and Walker (2005), Access & Use Information — Public
Denial of responsibility! Techno Blender is an automatic aggregator of the all world’s media. In each content, the hyperlink to the primary source is specified. All trademarks belong to their rightful owners, all materials to their authors. If you are the owner of the content and do not want us to publish your materials, please contact us by email – [email protected]. The content will be deleted within 24 hours.