How to get the condition of OLS mathematically?

Question

From this discussion, I asked @tdm about the condition of OLS but I still cannot get it, and it is not easy to answer the mathematic equation in comment part so I want to ask here.

One way to see this is to notice that OLS minimizes

$$\frac{1}{n} \sum_i (y_i - \alpha - \beta x)^2$$

with respect to $\alpha$ and $\beta$ I do not understand clearly why it leads to the conditions

$$\frac{1}{n} \sum_i (y_i - \alpha - \beta x) = 0$$ and

$$\frac{1}{n} \sum_i (y_i - \alpha - \beta x) x = 0$$

Answer 1

Your goal is to find a line that has a best linear fit through set of observations $y,x$. Every equation of a line in 2D space will have two parameters, slope $\beta$ and intercept $\alpha$, so your problem boils down to choose $\alpha$ and $\beta$ that best fit the data.

It is reasonable to define the best fit as a fit that minimizes the sum of squared residuals. We want to minimizes the sum of residuals because we want errors to be as small as possible, and we square them to ensure that positive errors do not cancel with negative ones.

As a consequence we minimize the following objective function

$$\min_{\alpha, \beta} \frac{1}{n} \sum \epsilon^2 = \min_{\alpha, \beta} \frac{1}{n} \sum (y - \alpha - \beta x)^2$$

wrt $\alpha$ and $\beta$. In order to find a minimum of a function you have to calculate first order conditions which are given at a points where first derivatives equal zero.

$$ \frac{d\epsilon}{d\alpha}= 0 \implies -2\frac{1}{n} \sum(y - \alpha - \beta x) = 0$$

$$\frac{d\epsilon}{d\beta} = 0 \implies -2\frac{1}{n} \sum (y- \alpha - \beta x) x = 0$$

where -2 and actually technically even $\frac{1}{n}$ can be dropped since you can just divide both sides by them and they disappear, but $\frac{1}{n}$ is usually left as then you can rewrite the optimum solution for $\beta$ in terms of variances and covariances as tdm does in his excellent answer.