# Finding the distribution in OLS

I have the following linear regression model: $$Y_i = \beta_0 + \beta_1 x_i + \epsilon_i$$ where $$\epsilon_i$$ are independent $$N(0, \sigma^2)$$ random variables. Let $$\hat{\beta}_i$$ denote the ordinary least squares estimators of the parameters $$\beta_0$$ and $$\beta_1$$ for $$i=0,1$$. I want to determine the distribution of $$\frac{\hat{\beta}_0 - \beta_0}{\sqrt{s^2 \left( \frac{1}{n} + \frac{\bar{x}^2}{s_{xx}} \right)}}$$ where $$s^2 = \frac{1}{n-2} \left( s_{yy} - \frac{s_{xy}^2}{s_{xx}} \right)$$ and $$s_{xx}$$, $$s_{yy}$$, and $$s_{xy}$$ are the sums of squares and cross-products.

Additionally, I want to derive a $$(1-\alpha)100\%$$ confidence interval for $$\beta_0$$. Could someone help me with the steps and derivation?

The distribution will be $$t_{n-2}$$.

Below, I assume the sample $$\{x_i\}_{i=1}^n$$ is treated as fixed. That is, I do all computations as if we were conditioning on $$\{x_i\}_{i=1}^n$$. I relax this assumption at the end.

I think the easiest way to get the distribution of the regressors is by first transitioning the setup into matrix notation. This avoids (most of) the need for partitioned regression. That is, we write $$Y=X\beta+\varepsilon$$ where $$Y=\begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}, \qquad X=\begin{bmatrix} 1 & x_1 \\ 1 & x_2 \\ \vdots & \vdots \\ 1 & x_n \end{bmatrix}, \qquad \beta = \begin{bmatrix} \beta_1 & \beta_1 & \dots \ \ \beta_1 \\ \beta_2 & \beta_2 & \dots \ \ \beta_2 \end{bmatrix}, \qquad \varepsilon=\begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix}$$ and $$\varepsilon \sim N(0,\sigma^2I_n)$$ since $$\{\epsilon_i\}_{i=1}^n$$ is an i.i.d. sequence of normal random variables.

In this notation, our OLSE is $$\hat{\beta}=(X'X)^{-1}X'Y$$. Plugging in $$Y=X\beta +\varepsilon$$ we find $$\hat{\beta}=(X'X)^{-1}X'(X\beta+\varepsilon)=\beta+(X'X)^{-1}X'\varepsilon$$

Therefore, $$\hat{\beta}-\beta=(X'X)^{-1}X'\varepsilon$$. Hence, since the sample $$\{x_i\}_{i=1}^n$$ is treated as fixed and $$\varepsilon \sim N(0,\sigma^2 I_n)$$, we have $$\begin{pmatrix} \hat{\beta}_0 \\ \hat{\beta}_1 \end{pmatrix}-\begin{pmatrix} \beta_0 \\ \beta_1 \end{pmatrix}=\hat{\beta}-\beta\sim N(0,\sigma^2 (X'X)^{-1})$$

Any linear combination of jointly normally distributed random variables is normal. Define $$\lambda=\begin{bmatrix} 1 \\ 0 \end{bmatrix}$$, then $$\hat{\beta}_0-\beta_0 =\lambda'(\hat{\beta}-\beta)\sim N(0,\sigma^2\lambda'(X'X)^{-1}\lambda)$$ Therefore, $$\frac{\hat{\beta}_0-\beta_0}{\sqrt{\sigma^2\lambda'(X'X)^{-1}\lambda}}\sim N(0,1)$$ Consider, \begin{align*} \lambda'(X'X)^{-1} \lambda &= \lambda' \begin{bmatrix} n & \sum_{i=1}^n x_i \\ \sum_{i=1}^n x_i & \sum_{i=1}^n x_i^2\end{bmatrix}^{-1} \lambda \\ &= \frac{1}{n\sum_{i=1}^n x_i^2 -\left(\sum_{i=1}^n x_i \right)^2} \lambda' \begin{bmatrix} \sum_{i=1}^n x_i^2 & -\sum_{i=1}^n x_i \\ -\sum_{i=1}^n x_i & n \end{bmatrix}\lambda \\ &= \frac{\sum_{i=1}^n x_i^2}{n\sum_{i=1}^n x_i^2 -\left(\sum_{i=1}^n x_i \right)^2} \\ &= \frac{1}{n}\left[\frac{\frac{1}{n}\sum_{i=1}^n x_i^2}{\frac{1}{n}\sum_{i=1}^n x_i^2 -\left(\frac{1}{n}\sum_{i=1}^n x_i \right)^2}\right] \\ &= \frac{1}{n}\left[\frac{\frac{1}{n}\sum_{i=1}^n x_i^2}{\frac{1}{n}\sum_{i=1}^n x_i^2 -\bar{x}^2}\right] \\ &= \frac{1}{n}\left[\frac{\frac{1}{n}\sum_{i=1}^n x_i^2-\bar{x}^2+\bar{x}^2}{\frac{1}{n}\sum_{i=1}^n x_i^2 -\bar{x}^2}\right] \\ &= \frac{1}{n}\left[1+ \frac{\bar{x}^2}{\frac{1}{n}\sum_{i=1}^n x_i^2 -\bar{x}^2}\right] \\ &=\frac{1}{n}\left[1+ \frac{\bar{x}^2}{\frac{1}{n}\sum_{i=1}^n (x_i -\bar{x})^2}\right] \\ &= \frac{1}{n}+\frac{\bar{x}^2}{s_{xx}} \end{align*}

Hence, $$\frac{\hat{\beta}_0-\beta}{\sqrt{\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}\right)}}\sim N(0,1)$$

What we want to do now is remove $$\sigma^2$$ from our expression since it is unknown. For this, we can use the standard t-statistic approach.

Notice your expression for $$s^2$$ can be expressed in terms of the residuals $$\{\hat{\varepsilon}_i\}_{i=1}^n$$. Using that $$\hat{\beta}_1=\frac{s_{xy}}{s_{xx}}$$ (by partitioned regression), and $$\hat{\beta}_0=\bar{y}-\hat{\beta}_1\bar{x}$$, we have \begin{align*} \sum_{i=1}^n \hat{\varepsilon}^2_i &= \sum_{i=1}^n (y_i-\hat{y}_i)^2 \\ &=\sum_{i=1}^n(y_i-(\hat{\beta}_0+\hat{\beta}_1x_i))^2 \\ &=\sum_{i=1}^n(y_i-(\bar{y}-\frac{s_{xy}}{s_{xx}}\bar{x}+\frac{s_{xy}}{s_{xx}}x_i))^2 \\ &= \sum_{i=1}^n ((y_i-\bar{y})-\frac{s_{xy}}{s_{xx}}(x_i-\bar{x}))^2 \\ &= s_{yy}-2\frac{s_{xy}^2}{s_{xx}}+\frac{s_{xy}^2}{s_{xx}} \\ &=s_{yy}-\frac{s_{xy}^2}{s_{xx}}\end{align*} This is advantageous as residuals are the projection of the errors (whose distribution we do know) onto the space orthogonal to the regressors. That is, we can use the annihilator matrix $$M=I_n-X'(X'X)^{-1}X$$, which is the projection matrix onto the space $$\text{col}(X)^{\perp}$$, to relate the residuals to the errors. This matrix is useful for us as $$M\varepsilon=\hat{\varepsilon}$$. We can calculate, $$\text{tr}(M)=n-\text{tr}(X'(X'X)^{-1}X)=n-\text{tr}((X'X)^{-1}(X'X))=n-2$$ Therefore, since $$\frac{\varepsilon}{\sigma}\sim N(0,I_n)$$, by Cochran's theorem $$\frac{1}{\sigma^2}\sum_{i=1}^n \hat{\epsilon}_i=\frac{1}{\sigma^2}\hat{\varepsilon}'\hat{\varepsilon}=\frac{\varepsilon'}{\sigma} M \frac{\varepsilon}{\sigma}\sim \chi_{n-2}$$ So, $$(n-2) \frac{s^2}{\sigma^2}\sim \chi_{n-2}$$ Additionally, $$\hat{\beta}_0$$ and $$\varepsilon$$ are independent (see this question). Therefore, the two statistics \begin{align*} T_1 &=\frac{\hat{\beta}_0-\beta}{\sqrt{\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}\right)}} \sim N(0,1) \\ T_2 &= (n-2) \frac{s^2}{\sigma^2}\sim \chi_{n-2}\end{align*} are independent. Therefore, $$\textit{by definition}$$ we have $$\frac{T_1}{\sqrt{T_2/(n-2)}}\sim t_{n-2}$$ where $$t_{n-2}$$ is the t-distribution with $$n-2$$ degrees of freedom. Cancelling terms gives $$\frac{\hat{\beta}_0-\beta_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}\right)}}\sim t_{n-2}$$ this is not normal as we have to account for the estimation error of $$\sigma^2$$ by $$s^2$$.

We can conclude something more. Everything I have done has been implicitly conditioning on the sample and treating it as if it were the population. This could be a problematic assumption. However, since the distribution, $$t_{n-2}$$, does not depend on the give sample data, we can remove the conditioning on $$\{x_i\}_{i=1}^n$$. Therefore, even for stochastic regressors we have $$\frac{\hat{\beta}_0-\beta_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}\right)}}\sim t_{n-2}$$ so we do not need to worry about whether we conditioning on the sample or not for the result.

To derive a two-sided confidence interval of size $$1-\alpha$$, let the $$\frac{\alpha}{2}$$ and $$1-\frac{\alpha}{2}$$ quantiles of the $$t_{n-2}$$ distribution be $$t_{\frac{\alpha}{2},n-2}$$ and $$t_{1-\frac{\alpha}{2},n-2}$$, respectively. Then, defining $$\hat{se}(\hat{\beta}_0)=\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{xx}}\right)}$$, we have $$1-\alpha = \mathbb{P}\left(t_{\frac{\alpha}{2}}\leq \frac{\hat{\beta}_0-\beta_0}{\hat{se}(\hat{\beta}_0)} \leq t_{1-\frac{\alpha}{2},n-2} \right)$$ Rearranging, we obtain $$1-\alpha =\mathbb{P}\left(\hat{\beta}_0-\hat{se}(\hat{\beta}_0)t_{\frac{\alpha}{2},n-2}\geq \beta_0 \geq \hat{\beta}_0-\hat{se}(\hat{\beta}_0)t_{1-\frac{\alpha}{2},n-2} \right)$$ Therefore, $$CI(1-\alpha)=\left[\hat{\beta}_0-\hat{se}(\hat{\beta}_0)t_{1-\frac{\alpha}{2},n-2},\hat{\beta}_0-\hat{se}(\hat{\beta}_0)t_{\frac{\alpha}{2},n-2}\right]$$

The variance of $$\hat{\beta}_0$$ is given by: $$\operatorname{Var}\left(\hat{\beta}_0\right)=\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)$$ where $$s_{x x}=\sum_{i=1}^n\left(x_i-\bar{x}\right)^2$$. The estimator $$\hat{\beta}_0$$ is normally distributed because the errors $$\epsilon_i$$ are normally distributed. Hence: $$\hat{\beta}_0 \sim N\left(\beta_0, \sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)\right) .$$

To standardize $$\hat{\beta}_0$$, we consider: $$\frac{\hat{\beta}_0-\beta_0}{\sqrt{\operatorname{Var}\left(\hat{\beta}_0\right)}}=\frac{\hat{\beta}_0-\beta_0}{\sqrt{\sigma^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s x x}\right)}} \sim N(0,1) \text {. }$$

Since we do not know $$\sigma^2$$, we replace it with the unbiased estimator $$s^2$$, where: $$s^2=\frac{1}{n-2}\left(s_{y y}-\frac{s_{x y}^2}{s_{x x}}\right)$$ and $$s_{y y}=\sum_{i=1}^n\left(Y_i-\bar{Y}\right)^2, s_{x y}=\sum_{i=1}^n\left(x_i-\bar{x}\right)\left(Y_i-\bar{Y}\right)$$. Thus, the standardized form becomes: $$\frac{\hat{\beta}_0-\beta_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)}} .$$

Under the assumption of normally distributed errors, this follows a $$t$$-distribution with $$n-2$$ degrees of freedom: $$\frac{\hat{\beta}_0-\beta_0}{\sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)}} \sim t_{n-2}$$

A $$(1-\alpha) 100 \%$$ confidence interval for $$\beta_0$$ is derived from the $$t$$-distribution: $$\hat{\beta}_0 \pm t_{\alpha / 2, n-2} \cdot \sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)}$$ where $$t_{\alpha / 2, n-2}$$ is the critical value of the $$t$$-distribution with $$n-2$$ degrees of freedom at the $$\alpha / 2$$ significance level.

Thus, the confidence interval for $$\beta_0$$ is: $$\left[\hat{\beta}_0-t_{\alpha / 2, n-2} \cdot \sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)}, \hat{\beta}_0+t_{\alpha / 2, n-2} \cdot \sqrt{s^2\left(\frac{1}{n}+\frac{\bar{x}^2}{s_{x x}}\right)}\right] .$$

This provides the desired $$(1-\alpha) 100 \%$$ confidence interval for $$\beta_0$$.