# Sum of residuals and Matrix calculus (step by step)

1. Write the sum of residuals in matrix form.

Attempt: $(y-X\boldsymbol{\beta})^T(y-X\boldsymbol{\beta}) = y^Ty-\boldsymbol{\beta}^TX^Ty-y^TX\boldsymbol{\beta}+\boldsymbol{\beta}^TX^TX\boldsymbol{\beta} = y^Ty-2y^TX\boldsymbol{\beta}+\boldsymbol{\beta}^TX^TX\boldsymbol{\beta}.$

1. Minimize this and verify that $\hat{\boldsymbol{\beta}} = (X^TX)^{-1}X^Ty$.

Attempt: $\frac{\partial SSR(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} = -2y^TX+2X^TX\boldsymbol{\beta} = 0 \rightarrow X^TX\boldsymbol{\beta}=y^TX \rightarrow \hat{\boldsymbol{\beta}} = (X^TX)^{-1}y^TX = (X^TX)^{-1}X^Ty.$

I haven't studied matrix calculus, so I am a little bit confused about the step in calculation. In the first problem, could you show me which rule is used for the first equality in the equation. And in the second problem, I am a little confused about the first step of differentiation.

I really appreciate if you show the calculation step by step.

Let $\mathbf A,\mathbf B$ be $m\times n$ matrices (or vectors is either $m$ or $n$ is $1$), \begin{align} (\mathbf A+\mathbf B)^T&=\mathbf A^T+\mathbf B^T \tag{1}\\ (\mathbf A\mathbf B)^T&=\mathbf B^T\mathbf A^T \tag{2} \end{align}
Thus, \begin{align} (\mathbf y-\mathbf X\beta)^T(\mathbf y-\mathbf X\beta)&=(\mathbf y^T-(\mathbf X\beta)^T)(\mathbf y-\mathbf X\beta)\\ &=(\mathbf y^T-\beta^T\mathbf X^T)(\mathbf y-\mathbf X\beta)\\ &=\mathbf y^T(\mathbf y-\mathbf X\beta)-\beta^T\mathbf X^T(\mathbf y-\mathbf X\beta)\\ &=\mathbf y^T\mathbf y-\mathbf y^T\mathbf X\beta-\beta^T\mathbf X^T\mathbf y+\beta^T\mathbf X^T\mathbf X\beta \\ &=\mathbf y^T\mathbf y-2\mathbf y^T\mathbf X\beta+\beta^T\mathbf X^T\mathbf X\beta \tag{3} \end{align} where line 1 uses $(1)$, line 2 uses $(2)$, and the last line follows from the fact that $\mathbf y^T\mathbf X\beta$ and $\beta^T\mathbf X^T\mathbf y$ are both $1\times1$ scalars.
Let $\mathbf a,\mathbf x$ be $n\times 1$ vectors and $\mathbf A$ a symmetric $n\times n$ matrix. Matrix calculus has the following rules: \begin{align} \frac{\mathrm d\ \mathbf a^T\mathbf z}{\mathrm d\ \mathbf z}&=\mathbf a \tag{4} \\ \frac{\mathrm d\ \mathbf z^T\mathbf A\mathbf z}{\mathrm d\ \mathbf z}&=2\mathbf A\mathbf z \tag{5} \end{align}
Applying these to differentiating $(3)$ with respect to $\mathbf\beta$, we get the first order condition: $$-2\mathbf X^T\mathbf y+2\mathbf X^T\mathbf X\beta=0$$ Solving for $\beta$, we get the usual OLS formula.