3
$\begingroup$

There is a sample of $n$ observations, each element has a numeric $Y$ and $X$ characteristic. There is an OLS regression over the sample $$ Y = b_0 + b_1 X + \textbf{u}, $$ $\textbf{u}$ being the vector of residuals. Suppose we remove observation $i$ from the sample, restricting it to $n-1$ observations, and run an OLS regression again, yielding $$ Y_{-i} = b_0' + b_1' X_{-i} + \textbf{u}'. $$ Subscript $_{-i}$ denotes that $Y$ and $X$ are not the same vectors as before, as observation $i$ is missing.

Seems to me that if the observation $i$ we removed is on the original sample's 'regression line', that is if $$ Y_i = b_0 + b_1 X_i, $$ then $b_0 = b_0'$ and $b_1 = b_1'$.

Example: (in R code)

x = c(5,3,4,5,4,4)
y = c(20,15,14,21,10,25)

plot(x,y)
abline(coef(reg))

regression line

plot(x[-1],y[-1])

reg1 = lm(y[-1] ~ x[-1])
abline(coef(reg1))

subsample regression

My questions are:

1) Is this true, and if yes, what is the proof?
[I figured this one out in the meantime, but feel free to give a nicer solution.]

2) Is it possible that after removing an observation (not on the 'regression line') we have

($b_0 = b_0'$ and $b_1 \neq b_1'$) or ($b_0 \neq b_0'$ and $b_1 = b_1'$)?

$\endgroup$
2
$\begingroup$

Since Q1 has been solved, I'll focus on Q2. Yes, it's possible to remove a sample point that is not on the regression line but still yields $b_{1}=b_{1}'$ and $b_{0}\ne b_{0}'$. Such points have the property $x_{j}=\bar{x}$, where $\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_{i}$. These points are known as points without leverage.

Suppose $\left(x_{j},y_{j}\right)$ is a point without leverage but not on the regression line, that is, $x_{j}=\overline{x}$ and $y_{j}\ne\overline{y}$. We know

\begin{align*} b_{1} & =\frac{\sum_{i}\left(x_{i}-\bar{x}\right)y_{i}}{\sum_{i}\left(x_{i}-\bar{x}\right)^{2}}\\ & =\frac{\sum_{-j}\left(x_{i}-\bar{x}\right)y_{i}+\left(x_{j}-\overline{x}\right)y_{j}}{\sum_{-j}\left(x_{i}-\bar{x}\right)^{2}+\left(x_{j}-\overline{x}\right)^{2}}\\ & =\frac{\sum_{-j}\left(x_{i}-\bar{x}\right)y_{i}}{\sum_{-j}\left(x_{i}-\bar{x}\right)^{2}}\\ & =\frac{\sum_{-j}\left(x_{i}-\bar{x}_{-j}\right)y_{i}}{\sum_{-j}\left(x_{i}-\bar{x}_{-j}\right)^{2}}\\ & =b_{1}' \end{align*} here we use the conditions $x_{j}-\overline{x}=0$ and $\overline{x}=\overline{x}_{-j}$. Combined with the condition $\overline{y}\ne\overline{y}_{-j}$, we have \begin{align*} b_{0}' & =\overline{y}_{-j}-b_{1}'\overline{x}_{-j}\\ & \ne\overline{y}-b_{1}\overline{x}\\ & =b_{0} \end{align*}

$\endgroup$
2
$\begingroup$

I think I figured out 1). I am still interested in 2).

OLS sets $b_0$ and $b_1$ in such a way that

$$ (b_0 \ \ b_1) = \arg\min_{a_0, a_1} \sum_j \left(Y_j - a_0 - a_1 X_j\right)^2. $$ The first order conditions are $$ (-2)\sum_j \left(Y_j - b_0 - b_1 X_j\right) = 0 $$ and $$ (-2)\sum_j X_j \left(Y_j - b_0 - b_1 X_j\right) = 0. $$ If observation $i$ is on the regression line then $$ \begin{align*} \left(Y_i - b_0 - b_1 X_i\right) & = 0 \\ \\ X_i\left(Y_i - b_0 - b_1 X_i\right) & = 0. \end{align*} $$ It follows from this and the original first order conditions that $$ \begin{align*} (-2)\sum_{j\neq i} \left(Y_j - b_0 - b_1 X_j\right) & = 0 \\ \\ (-2)\sum_{j\neq i} X_j\left(Y_j - b_0 - b_1 X_j\right) & = 0, \end{align*} $$ which are the first order conditions of the subsample's OLS problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.