# Calculating standard error

Consider a regression model on the form:

$$y_{i} = \alpha + \beta_{1}X_{i} + u_{i}$$ (1)

I am given $$var(u_{i}|X_{i}) = 10$$ and I know the OLS estimates for $$\alpha$$ and $$\beta$$ and using this I have to calculate the standard errors of the parameter estimates.

I am following Hayashi's book and I know that

$$se(b_{k}) = \sqrt(s^{2}(X'X)^{-1}_{kk})$$

Where $$b_{k}$$ is the parameter estimate of interest. I know $$(X'X)^{-1}$$ but what is confusing me is going from the given model to expressing it in matrix notation and that this does not use the variance of the error term.

I can express (1) as $$Y = X\beta + u$$ and since $$var(u_{i}|X_{i}) = var(y_{i} - \alpha - \beta_{2}X_{i}|X_{i})$$ but is this equal to $$var(Y - X\beta|X)$$?

If so, I can express $$var(u_{i}|X_{i}) = 10$$ as $$\sigma^{2}I_{n}$$ where $$\sigma^{2} = 10$$. And then could I use the fact that

$$var(b_{k}|X) = (X'X)^{-1}X' \sigma^{2} I_{n} X(X'X)^{-1}$$

To find the standard errors of interest.

Your notation is a bit all over the place, so I'm going to try and standardize it for a general case.

Let $$X$$ be the regression matrix (which includes a column of 1s for the intercept term) and $$\beta$$ be the vector of coefficients to be estimated via OLS (which includes the intercept term). $$Var(u_i|X) = 10 = \sigma^2$$, for all $$i$$. Thus the errors are homoscedastic (constant variance) and the variance-covariance matrix of the errors $$\Omega$$ has diagonal entries that are all equal to $$\sigma^2$$. Furthermore, if you assume that the error terms are not serially correlated, then the off-diagonal covariance terms $$Cov(u_i, u_j|X)$$ for all $$i \neq j$$ in the variance-covariance matrix $$\Omega$$ are zero.

Those are two of the standard Gauss-Markov assumptions used to establish the BLUEness of the OLS estimator. Under these assumptions, $$\Omega$$ is

$$E[uu'|X] = \left[ \begin{array}{ccccc} \sigma^2 \\ & \sigma^2 & & \huge0 \\ & & \ddots \\ & \huge0 & & \sigma^2 \\ & & & & \sigma^2 \end{array} \right] = \sigma^2I.$$

To get the variance of the OLS estimates $$b = \hat{\beta}$$, first note that

\begin{align} b &= (X'X)^{-1}X'y \\ &= (X'X)^{-1}X'(X\beta + u) \\ &= \beta + (X'X)^{-1}X'u \\ \implies b - \beta &= (X'X)^{-1}X'u, \end{align}

using $$y = X\beta + u$$. Then,

\begin{align} Var(b|X) &= E[(b-\beta)(b-\beta)'|X] \\ &= E[(X'X)^{-1}X'u((X'X)^{-1}X'u)'|X] \\ &= E[(X'X)^{-1}X'uu'X(X'X)^{-1}|X] \\ &= (X'X)^{-1}X'E[uu'|X]X(X'X)^{-1}. \end{align}

But recall that we derived $$E[uu'|X] = \sigma^2 I$$ via the Gauss-Markov assumption of spherical errors. Thus, by substitution,

\begin{align} Var(b|X) &= (X'X)^{-1}X'E[uu'|X]X(X'X)^{-1} \\ &= (X'X)^{-1}X'(\sigma^2 I)X(X'X)^{-1} \\ &= \sigma^2 (X'X)^{-1} X'X(X'X)^{-1} \\ \implies Var(b) &= \sigma^2 (X'X)^{-1}. \end{align}

The standard deviation of $$b$$ is just the square root of the variance, or

$$sd(b) = \sqrt{\sigma^2 (X'X)^{-1}}.$$

To find the standard deviation of the $$k^{th}$$ estimated coefficient in $$b$$, $$b_{k}$$, we simply extract the $$k^{th}$$ diagonal element of $$(X'X)^{-1}$$, denoted as $$(X'X)_{kk}^{-1}$$:

$$sd(b_k) = \sqrt{\sigma^2 (X'X)_{kk}^{-1}}.$$

$$\sigma^2$$ is the (common) variance of the errors and, unfortunately, this value is unobserved in our sample. In your question, however, it appears that this value is just given to you directly — it's $$10$$. In general, however, $$\sigma^2$$ must be estimated using the data. It turns out that given homoscedastic errors, an unbiased estimator of $$\sigma^2$$ is

$$s^2 = \frac{e'e}{n-P},$$

where $$n$$ is the number of observations, $$P$$ is the number of columns in $$X$$, and $$e = \hat{u}$$; that is, the vector of residuals $$y - Xb$$. $$s$$, incidentally, is called the "standard error of the regression", not to be confused with the standard errors of the OLS estimates.

Thus, the standard error of $$\mathbf{b_k}$$ is estimated as

$$\boldsymbol{\widehat{se}(b_k) = \sqrt{s^2 (X'X)_{kk}^{-1}}},$$

which is Hayashi's result.

• Thank you so much for the thorough explanation! Really cleared things up. Oct 27, 2018 at 13:44