Are the OLS Estimators Normally Distributed in a Linear Regression Model? | by Aaron Zhu | Nov, 2022

Justification for the Normality Assumption

We all know that the normality assumption is optional to compute unbiased estimates in a linear regression model. In this post, we will discuss if the OLS estimators in a linear regression model are normally distributed and what assumptions would be needed to draw the conclusion.

What are the OLS estimators in a linear regression model?

The OLS estimators (β^) are computed from a sample to estimate the population parameters (β) in a linear regression model.

The OLS estimators are random variables with probability distributions (i.e., the sampling distributions of the OLS estimators). The sampling distribution describes the possible values the OLS estimators could take on across different samples.

We know that the OLS estimators can be solved with a closed-form solution for a given sample.

We can think of this in this way, conational on X, it is very likely that Y could take different values in different samples. Therefore, the OLS estimators (β^) vary with the response values (Y).

With a little bit of math, we can compute the mean and variance of the OLS estimators.

Furthermore, we expect the OLS estimators to have the following properties,

• The OLS estimators are unbiased: the expected values of the OLS estimators are equal to the true population parameter values.
• The OLS estimators are consistent: as the sample size increases, the OLS estimators converge to the true population parameter values.
• The OLS estimators are BLUE (least variance among all linear unbiased estimators). Therefore, they are the best (efficient) estimators.

How to estimate the variance of the OLS estimators in a Linear Regression Model?

In the formula for the variance of the OLS estimators, σ2 is the variance of the error term, which is an unknown parameter in the population model. We typically estimate this value using the residuals in the sample data.

S2 (aka, the mean square error, MSE, or the residual variance) is the unbiased estimator of the variance of the error (σ2). S2 can be computed in the following formula.

Therefore, if we set up a more accurate linear regression model, the residuals tend to be closer to 0, the MSE (S2) tend to be smaller, and the variance of the sampling distribution for the OLS estimators would be smaller, then we would end up having more precise OLS estimators (i.e., tighter confidence intervals).

What is the motivation for assuming the OLS estimators are normally distributed in a Linear Regression Model?

At this point, we haven’t made any normality assumption in the linear regression model. All we know is the OLS estimators are random variables and how to compute their means and variances.

We also know normality assumption is NOT required to compute unbiased, consistent, BLUE OLS estimators. Then why are we motivated to assume the OLS estimators are normally distributed in a Linear Regression Model?

The answer is very simple. We can never estimate the true popular parameters from the sample data. Our estimates will always be off the mark. The normality assumption of the OLS estimators allows us to compute the p-values for hypothesis testing and construct reliable confidence intervals.

Why is it reasonable to assume the OLS estimators are normally distributed in a Linear Regression Model?

Often we make the assumption that the error term in a linear regression model is normally distributed. Then it implies that the OLS estimators are also normally distributed. We can prove this easily.

In the above equation, we have shown that the OLS estimators are linear combinations of the error terms. Therefore, the normality assumption (i.e., the errors are normally distributed) implies the OLS estimators are normally distributed.

If the sample size is sufficiently large, we don’t need to assume the error term is normally distributed. Because when the sample size is large enough, the central limit theorem kicks in and justifies that the OLS estimators are well approximated by a multivariate normal distribution regardless of the distributions of the error terms.

How to compute confidence intervals for OLS estimators in a Linear Regression Model?

From the sample data, we can estimate the population parameters (β) with the OLS estimators and estimate the standard deviation (also called standard error) of the sampling distribution of the OLS estimators using the residuals. If we assume the OLS estimators are normally distributed, we then can compute the confidence intervals using

For example, the 95% confidence interval indicates that we are 95% confident that the CI contains the true value of the population parameter (βi). In other words, if we repeat this process with many different samples, 95% of the time that the CI contains the true value.

What are the factors affecting the distribution of the OLS estimators?

First of all, let’s rewrite the variance of the OLS estimators in the following format. You can find its derivation here.

Obviously, the sample size plays a huge role in the distribution of the OLS estimators. As the sample size increase, the sampling distribution of the OLS estimators will be closer to the normal distribution, and the variance of the OLS estimators will be smaller, which means we have more precise OLS estimators.

In other words, the more data points we have in the sample, the more capable the model is to capture the relationship between X and Y, and the more precise the OLS estimators are.

Furthermore, as the variance of the Xi increase, the variance of the corresponding OLS estimator will decrease.

In other words, the more information the explanatory can provide (i.e., higher variance), the more precise we can estimate the true value of the parameter.

Conclusion

The sampling distribution of the OLS estimators will approximate a normal distribution if either we assume the errors are normally distributed or the sample size is sufficiently large.

As the sample size increases, we would expect the distribution of the OLS estimators to have smaller variances.

Also, as the variance of the Xi increase, the variance of the corresponding OLS estimator will tend to decrease.

If you would like to explore more posts related to Statistics, please check out my articles:

If you enjoy this article and would like to Buy Me a Coffee, please click here.

You can sign up for a membership to unlock full access to my articles, and have unlimited access to everything on Medium. Please subscribe if you’d like to get an email notification whenever I post a new article.

Justification for the Normality Assumption

We all know that the normality assumption is optional to compute unbiased estimates in a linear regression model. In this post, we will discuss if the OLS estimators in a linear regression model are normally distributed and what assumptions would be needed to draw the conclusion.

What are the OLS estimators in a linear regression model?

The OLS estimators (β^) are computed from a sample to estimate the population parameters (β) in a linear regression model.

The OLS estimators are random variables with probability distributions (i.e., the sampling distributions of the OLS estimators). The sampling distribution describes the possible values the OLS estimators could take on across different samples.

We know that the OLS estimators can be solved with a closed-form solution for a given sample.

We can think of this in this way, conational on X, it is very likely that Y could take different values in different samples. Therefore, the OLS estimators (β^) vary with the response values (Y).

With a little bit of math, we can compute the mean and variance of the OLS estimators.

Furthermore, we expect the OLS estimators to have the following properties,

• The OLS estimators are unbiased: the expected values of the OLS estimators are equal to the true population parameter values.
• The OLS estimators are consistent: as the sample size increases, the OLS estimators converge to the true population parameter values.
• The OLS estimators are BLUE (least variance among all linear unbiased estimators). Therefore, they are the best (efficient) estimators.

How to estimate the variance of the OLS estimators in a Linear Regression Model?

In the formula for the variance of the OLS estimators, σ2 is the variance of the error term, which is an unknown parameter in the population model. We typically estimate this value using the residuals in the sample data.

S2 (aka, the mean square error, MSE, or the residual variance) is the unbiased estimator of the variance of the error (σ2). S2 can be computed in the following formula.

Therefore, if we set up a more accurate linear regression model, the residuals tend to be closer to 0, the MSE (S2) tend to be smaller, and the variance of the sampling distribution for the OLS estimators would be smaller, then we would end up having more precise OLS estimators (i.e., tighter confidence intervals).

What is the motivation for assuming the OLS estimators are normally distributed in a Linear Regression Model?

At this point, we haven’t made any normality assumption in the linear regression model. All we know is the OLS estimators are random variables and how to compute their means and variances.

We also know normality assumption is NOT required to compute unbiased, consistent, BLUE OLS estimators. Then why are we motivated to assume the OLS estimators are normally distributed in a Linear Regression Model?

The answer is very simple. We can never estimate the true popular parameters from the sample data. Our estimates will always be off the mark. The normality assumption of the OLS estimators allows us to compute the p-values for hypothesis testing and construct reliable confidence intervals.

Why is it reasonable to assume the OLS estimators are normally distributed in a Linear Regression Model?

Often we make the assumption that the error term in a linear regression model is normally distributed. Then it implies that the OLS estimators are also normally distributed. We can prove this easily.

In the above equation, we have shown that the OLS estimators are linear combinations of the error terms. Therefore, the normality assumption (i.e., the errors are normally distributed) implies the OLS estimators are normally distributed.

If the sample size is sufficiently large, we don’t need to assume the error term is normally distributed. Because when the sample size is large enough, the central limit theorem kicks in and justifies that the OLS estimators are well approximated by a multivariate normal distribution regardless of the distributions of the error terms.

How to compute confidence intervals for OLS estimators in a Linear Regression Model?

From the sample data, we can estimate the population parameters (β) with the OLS estimators and estimate the standard deviation (also called standard error) of the sampling distribution of the OLS estimators using the residuals. If we assume the OLS estimators are normally distributed, we then can compute the confidence intervals using

For example, the 95% confidence interval indicates that we are 95% confident that the CI contains the true value of the population parameter (βi). In other words, if we repeat this process with many different samples, 95% of the time that the CI contains the true value.

What are the factors affecting the distribution of the OLS estimators?

First of all, let’s rewrite the variance of the OLS estimators in the following format. You can find its derivation here.

Obviously, the sample size plays a huge role in the distribution of the OLS estimators. As the sample size increase, the sampling distribution of the OLS estimators will be closer to the normal distribution, and the variance of the OLS estimators will be smaller, which means we have more precise OLS estimators.

In other words, the more data points we have in the sample, the more capable the model is to capture the relationship between X and Y, and the more precise the OLS estimators are.

Furthermore, as the variance of the Xi increase, the variance of the corresponding OLS estimator will decrease.

In other words, the more information the explanatory can provide (i.e., higher variance), the more precise we can estimate the true value of the parameter.

Conclusion

The sampling distribution of the OLS estimators will approximate a normal distribution if either we assume the errors are normally distributed or the sample size is sufficiently large.

As the sample size increases, we would expect the distribution of the OLS estimators to have smaller variances.

Also, as the variance of the Xi increase, the variance of the corresponding OLS estimator will tend to decrease.

If you would like to explore more posts related to Statistics, please check out my articles:

If you enjoy this article and would like to Buy Me a Coffee, please click here.

You can sign up for a membership to unlock full access to my articles, and have unlimited access to everything on Medium. Please subscribe if you’d like to get an email notification whenever I post a new article.