2
$\begingroup$
  1. Write the sum of residuals in matrix form.

Attempt: $(y-X\boldsymbol{\beta})^T(y-X\boldsymbol{\beta}) = y^Ty-\boldsymbol{\beta}^TX^Ty-y^TX\boldsymbol{\beta}+\boldsymbol{\beta}^TX^TX\boldsymbol{\beta} = y^Ty-2y^TX\boldsymbol{\beta}+\boldsymbol{\beta}^TX^TX\boldsymbol{\beta}.$

  1. Minimize this and verify that $\hat{\boldsymbol{\beta}} = (X^TX)^{-1}X^Ty$.

Attempt: $\frac{\partial SSR(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -2y^TX+2X^TX\boldsymbol{\beta} = 0 \rightarrow X^TX\boldsymbol{\beta}=y^TX \rightarrow \hat{\boldsymbol{\beta}} = (X^TX)^{-1}y^TX = (X^TX)^{-1}X^Ty.$

I haven't studied matrix calculus, so I am a little bit confused about the step in calculation. In the first problem, could you show me which rule is used for the first equality in the equation. And in the second problem, I am a little confused about the first step of differentiation.

I really appreciate if you show the calculation step by step.

$\endgroup$
5
$\begingroup$

Let $\mathbf A,\mathbf B$ be $m\times n$ matrices (or vectors is either $m$ or $n$ is $1$), \begin{align} (\mathbf A+\mathbf B)^T&=\mathbf A^T+\mathbf B^T \tag{1}\\ (\mathbf A\mathbf B)^T&=\mathbf B^T\mathbf A^T \tag{2} \end{align}

Thus, \begin{align} (\mathbf y-\mathbf X\beta)^T(\mathbf y-\mathbf X\beta)&=(\mathbf y^T-(\mathbf X\beta)^T)(\mathbf y-\mathbf X\beta)\\ &=(\mathbf y^T-\beta^T\mathbf X^T)(\mathbf y-\mathbf X\beta)\\ &=\mathbf y^T(\mathbf y-\mathbf X\beta)-\beta^T\mathbf X^T(\mathbf y-\mathbf X\beta)\\ &=\mathbf y^T\mathbf y-\mathbf y^T\mathbf X\beta-\beta^T\mathbf X^T\mathbf y+\beta^T\mathbf X^T\mathbf X\beta \\ &=\mathbf y^T\mathbf y-2\mathbf y^T\mathbf X\beta+\beta^T\mathbf X^T\mathbf X\beta \tag{3} \end{align} where line 1 uses $(1)$, line 2 uses $(2)$, and the last line follows from the fact that $\mathbf y^T\mathbf X\beta$ and $\beta^T\mathbf X^T\mathbf y$ are both $1\times1$ scalars.


Let $\mathbf a,\mathbf x$ be $n\times 1$ vectors and $\mathbf A$ a symmetric $n\times n$ matrix. Matrix calculus has the following rules: \begin{align} \frac{\mathrm d\ \mathbf a^T\mathbf z}{\mathrm d\ \mathbf z}&=\mathbf a \tag{4} \\ \frac{\mathrm d\ \mathbf z^T\mathbf A\mathbf z}{\mathrm d\ \mathbf z}&=2\mathbf A\mathbf z \tag{5} \end{align}

Applying these to differentiating $(3)$ with respect to $\mathbf\beta$, we get the first order condition: \begin{equation} -2\mathbf X^T\mathbf y+2\mathbf X^T\mathbf X\beta=0 \end{equation} Solving for $\beta$, we get the usual OLS formula.

$\endgroup$
  • $\begingroup$ Thanks. Can you recommend any source to see the proof of rules that you mentioned for the differentiation? $\endgroup$ – shk910 Aug 3 '18 at 4:07
  • $\begingroup$ @SihyunKim the Wikipedia entry on matrix calculus is a good start. $\endgroup$ – Herr K. Aug 3 '18 at 4:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.