3
$\begingroup$

In deriving our ordinary least squares estimates, we can partially differentiate the sum of squared errors $\sum_{i=1}^{n} {e_i^2} = \sum_{i=1}^{n} {(Y_i- \hat{\alpha}-\hat{\beta}X_i )^2}$ with respect to our estimators $\hat{\alpha}$ and $\hat{\beta}$ to get the following first order conditions:

$$\frac{\delta SSR}{\delta \hat{\alpha}}= -2\sum_{i=1}^{n} {e_i}=-2\sum_{i=1}^{n} {(Y_i- \hat{\alpha}-\hat{\beta}X_i)}=0$$ $$\frac{\delta SSR}{\delta \hat{\beta}}= -2\sum_{i=1}^{n} {X_i e_i}=-2\sum_{i=1}^{n} X_i {(Y_i- \hat{\alpha}-\hat{\beta}X_i)}=0$$

How can we then prove that the sum of squared residuals is a global minimum?

Firstly, can I confirm that the Hessian looks like this?

\begin{array}{cc} 2n & 2\sum_{i=1}^{n}X_i \\ 2\sum_{i=1}^{n}X_i & 2\sum_{i=1}^{n}(X_i)^2 \\ \end{array}

Secondly, how can I show that the determinant of this Hessian is positive definite, i.e. $ 4[n \sum_{i=1}^{n}(X_i)^2 - (\sum_{i=1}^{n}X_i)^2]>0?$ The presentation bears some resemblance to the population variance formula, and I know that variance must be non-negative, but I'm wondering how to move on. Is there some way to factorise this? Thank you.

$\endgroup$
3
$\begingroup$

It seems that you working on the right lines. By multiplying out the brackets, it can easily be verified that $n\sum{X}_i^2 - (\sum{X}_i)^2 = n\sum{(X_i - \bar{X}})^2$ where $\bar{X}$ is the mean. As you yourself spotted, this is basically a standard expression for the variance. From this, it follows that $n\sum{X}_i^2 - (\sum{X}_i)^2 > 0$ provided that $X_i \neq \bar{X}$ for some $i$.

Incidentally, you also need to verify that one of the second derivatives is positive (say $SSR{_{\alpha\alpha}}>0$) which follows directly from your Hessian.

$\endgroup$
  • $\begingroup$ The condition requires $n \sum_{i=1}^n X_i^2>(\sum_{i=1}^n X_i)^2$, not $n (\sum_{i=1}^n X_i)^2 > \sum_{i=1}^n X_i^2.$ $\endgroup$ – dlnB Mar 19 at 17:19
  • $\begingroup$ Good answer (+1) $\endgroup$ – dlnB Mar 19 at 17:34
1
$\begingroup$

The conditions you derive guarantee that $(\hat{\alpha}, \hat{\beta})$ occur where SSE is locally minimized. Since our estimates are unique, i.e. there is a unique parameter vector that satisfies our first-order conditions, we know the selected parameter vector minimizes the objective function in the interior of the parameter space.

To determine whether this is a global minimum, you would compare the SSE under our estimates to the boundary points. For a simple linear regression as you've described, the parameter space is $\mathbb{R}^2$, and is therefore unbounded so $(\hat{\alpha}, \hat{\beta})$ globally minimizes SSE. In contrast, if you had, for example, the restriction that a parameter was non-negative, checking the boundary condition would matter for determining a global minimum of SSE.

$\endgroup$
  • $\begingroup$ The fact that the first order conditions hold does not mean that we have a minimum - it might be a maximum! (Or neither.) This is why we need to take a look at the matrix of second derivatives $\endgroup$ – afreelunch Mar 19 at 16:59
  • $\begingroup$ When I said the conditions you derive, I meant the first-order conditions and the second-order conditions. $\endgroup$ – dlnB Mar 19 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.