From this discussion, I asked @tdm about the condition of OLS but I still cannot get it, and it is not easy to answer the mathematic equation in comment part so I want to ask here.

One way to see this is to notice that OLS minimizes

$$\frac{1}{n} \sum_i (y_i - \alpha - \beta x)^2$$

with respect to $\alpha$ and $\beta$ I do not understand clearly why it leads to the conditions

$$\frac{1}{n} \sum_i (y_i - \alpha - \beta x) = 0$$ and

$$\frac{1}{n} \sum_i (y_i - \alpha - \beta x) x = 0$$

  • 2
    $\begingroup$ This post has no question in it, please tell us what is your question $\endgroup$
    – 1muflon1
    Jun 19 at 0:08
  • $\begingroup$ Hi @1muflon1, sorry if my question is not clear to you, but the question is "why it leads to the conditions" as above. Thank you so much. $\endgroup$
    – Louise
    Jun 19 at 0:12
  • $\begingroup$ Ah, I will change the title to let it be as same as a question, thank you for your suggestion $\endgroup$
    – Louise
    Jun 19 at 0:14
  • 1
    $\begingroup$ @BeautifulMindset Do you know what an optimization problem is, and are you familiar with derivatives? $\endgroup$
    – Giskard
    Jun 19 at 6:51
  • $\begingroup$ @Giskard I am not familiar with optimization problem, but I am familiar with derivatives $\endgroup$
    – Louise
    Jun 19 at 7:45

Your goal is to find a line that has a best linear fit through set of observations $y,x$. Every equation of a line in 2D space will have two parameters, slope $\beta$ and intercept $\alpha$, so your problem boils down to choose $\alpha$ and $\beta$ that best fit the data.

It is reasonable to define the best fit as a fit that minimizes the sum of squared residuals. We want to minimizes the sum of residuals because we want errors to be as small as possible, and we square them to ensure that positive errors do not cancel with negative ones.

As a consequence we minimize the following objective function

$$\min_{\alpha, \beta} \frac{1}{n} \sum \epsilon^2 = \min_{\alpha, \beta} \frac{1}{n} \sum (y - \alpha - \beta x)^2$$

wrt $\alpha$ and $\beta$. In order to find a minimum of a function you have to calculate first order conditions which are given at a points where first derivatives equal zero.

$$ \frac{d\epsilon}{d\alpha}= 0 \implies -2\frac{1}{n} \sum(y - \alpha - \beta x) = 0$$

$$\frac{d\epsilon}{d\beta} = 0 \implies -2\frac{1}{n} \sum (y- \alpha - \beta x) x = 0$$

where -2 and actually technically even $\frac{1}{n}$ can be dropped since you can just divide both sides by them and they disappear, but $\frac{1}{n}$ is usually left as then you can rewrite the optimum solution for $\beta$ in terms of variances and covariances as tdm does in his excellent answer.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.