Calculating standard error

Question

Consider a regression model on the form:

$y_{i} = \alpha + \beta_{1}X_{i} + u_{i}$ (1)

I am given $var(u_{i}|X_{i}) = 10$ and I know the OLS estimates for $\alpha$ and $\beta$ and using this I have to calculate the standard errors of the parameter estimates.

I am following Hayashi's book and I know that

$se(b_{k}) = \sqrt(s^{2}(X'X)^{-1}_{kk})$

Where $b_{k}$ is the parameter estimate of interest. I know $(X'X)^{-1}$ but what is confusing me is going from the given model to expressing it in matrix notation and that this does not use the variance of the error term.

I can express (1) as $Y = X\beta + u$ and since $var(u_{i}|X_{i}) = var(y_{i} - \alpha - \beta_{2}X_{i}|X_{i})$ but is this equal to $var(Y - X\beta|X)$?

If so, I can express $var(u_{i}|X_{i}) = 10$ as $\sigma^{2}I_{n}$ where $\sigma^{2} = 10$. And then could I use the fact that

$var(b_{k}|X) = (X'X)^{-1}X' \sigma^{2} I_{n} X(X'X)^{-1}$

To find the standard errors of interest.

Answer 1

Your notation is a bit all over the place, so I'm going to try and standardize it for a general case.

Let $X$ be the regression matrix (which includes a column of 1s for the intercept term) and $\beta$ be the vector of coefficients to be estimated via OLS (which includes the intercept term). $Var(u_i|X) = 10 = \sigma^2$, for all $i$. Thus the errors are homoscedastic (constant variance) and the variance-covariance matrix of the errors $\Omega$ has diagonal entries that are all equal to $\sigma^2$. Furthermore, if you assume that the error terms are not serially correlated, then the off-diagonal covariance terms $Cov(u_i, u_j|X)$ for all $i \neq j$ in the variance-covariance matrix $\Omega$ are zero.

Those are two of the standard Gauss-Markov assumptions used to establish the BLUEness of the OLS estimator. Under these assumptions, $\Omega$ is

$$ E[uu'|X] = \left[ \begin{array}{ccccc} \sigma^2 \\ & \sigma^2 & & \huge0 \\ & & \ddots \\ & \huge0 & & \sigma^2 \\ & & & & \sigma^2 \end{array} \right] = \sigma^2I. $$

To get the variance of the OLS estimates $b = \hat{\beta}$, first note that

$$ \begin{align} b &= (X'X)^{-1}X'y \\ &= (X'X)^{-1}X'(X\beta + u) \\ &= \beta + (X'X)^{-1}X'u \\ \implies b - \beta &= (X'X)^{-1}X'u, \end{align} $$

using $y = X\beta + u$. Then,

$$ \begin{align} Var(b|X) &= E[(b-\beta)(b-\beta)'|X] \\ &= E[(X'X)^{-1}X'u((X'X)^{-1}X'u)'|X] \\ &= E[(X'X)^{-1}X'uu'X(X'X)^{-1}|X] \\ &= (X'X)^{-1}X'E[uu'|X]X(X'X)^{-1}. \end{align} $$

But recall that we derived $E[uu'|X] = \sigma^2 I$ via the Gauss-Markov assumption of spherical errors. Thus, by substitution,

$$ \begin{align} Var(b|X) &= (X'X)^{-1}X'E[uu'|X]X(X'X)^{-1} \\ &= (X'X)^{-1}X'(\sigma^2 I)X(X'X)^{-1} \\ &= \sigma^2 (X'X)^{-1} X'X(X'X)^{-1} \\ \implies Var(b) &= \sigma^2 (X'X)^{-1}. \end{align} $$

The standard deviation of $b$ is just the square root of the variance, or

$$ sd(b) = \sqrt{\sigma^2 (X'X)^{-1}}. $$

To find the standard deviation of the $k^{th}$ estimated coefficient in $b$, $b_{k}$, we simply extract the $k^{th}$ diagonal element of $(X'X)^{-1}$, denoted as $(X'X)_{kk}^{-1}$:

$$ sd(b_k) = \sqrt{\sigma^2 (X'X)_{kk}^{-1}}. $$

---

$\sigma^2$ is the (common) variance of the errors and, unfortunately, this value is unobserved in our sample. In your question, however, it appears that this value is just given to you directly — it's $10$. In general, however, $\sigma^2$ must be estimated using the data. It turns out that given homoscedastic errors, an unbiased estimator of $\sigma^2$ is

$$ s^2 = \frac{e'e}{n-P}, $$

where $n$ is the number of observations, $P$ is the number of columns in $X$, and $e = \hat{u}$; that is, the vector of residuals $y - Xb$. $s$, incidentally, is called the "standard error of the regression", not to be confused with the standard errors of the OLS estimates.

Thus, the standard error of $\mathbf{b_k}$ is estimated as

$$ \boldsymbol{\widehat{se}(b_k) = \sqrt{s^2 (X'X)_{kk}^{-1}}}, $$

which is Hayashi's result.