1
$\begingroup$

If I'm running a linear regression for example, and I take out some points, wouldn't the same line/plane still fit the data? If not, wouldn't that show that the data doesn't have a linear relationship?

Example: https://ocw.mit.edu/courses/economics/14-382-econometrics-spring-2017/lecture-notes/MIT14_382S17_lec1.pdf. On page 10, in the gender wage gap example, it finds a predictive effect of ~20% in the full sample and ~7% in the never married sample. But then if dWage/dGender depends on Marital Status, how can the functional form be right?

$\endgroup$
2
$\begingroup$

If I'm running a linear regression for example, and I take out some points, wouldn't the same line/plane still fit the data?

Aside from special cases no, OLS fits a line that is created based on minimization of sum of squared errors:

$$\min \sum e_i^2 = \min \sum (y_i-\hat{y_i})=\min \sum (y_i - \mathbf{x}_i^{\prime} \beta)^2 .$$

If you remove some observations then you will likely change the objective function above leading to some other line having the best fit through data (save for special cases like when all observations are on a single line).

If not, wouldn't that show that the data doesn't have a linear relationship?

Not necessarily, the linear relationship is assumed to exist between the variables in the 'background', but since we cannot directly observe the data generating process and there will always be some disturbance term even if we are sure there should be a linear relationship between the two variables the observations will not necessarily all fall on a single line.

Non-linearity cannot be detected just from the fact that line you fit changes when you remove some data (although that is an indicator of robustness of a result - robust results should not change too much when sample is expanded/cut slightly). However, you can detect non-linearity in other ways. For example, you can look for patterns in residual plot. If residual plot shows some curved pattern then that is an indicator of potential non-linearity. There are also rigorous tests for non-linearity (like Ramsey test etc) but to go over all of such ways would be beyond the scope of an SE answer - you can read more about them in econometrics textbooks such as Wooldridge Introductory Econometrics: A Modern Approach.

$\endgroup$
8
  • $\begingroup$ how to do the ramsey test of non-linearity $\endgroup$
    – develarist
    Nov 7 '20 at 16:38
  • $\begingroup$ @develarist that would require separate answer. If that is what you are interested in post it as a new question on the site. Or you can just google Ramsey test for non-linearity $\endgroup$
    – 1muflon1
    Nov 7 '20 at 16:40
  • $\begingroup$ it says Ramsey RESET test is for testing model mis-specification $\endgroup$
    – develarist
    Nov 7 '20 at 18:31
  • $\begingroup$ @develarist right but non-linearity is a form of mis-specification. If model is supposed to be $y= a + b_1 x_1 + b_2 x_1^2$ (which is non-linear) but you estimate $y = a + b_1 x_1$ (which is linear) you are mis-specifying your model. Hence the RESET test can test for non-linearity if properly set up $\endgroup$
    – 1muflon1
    Nov 7 '20 at 18:33
  • $\begingroup$ non-linearity in the variables or non-linearity in the coefficients? $\endgroup$
    – develarist
    Nov 7 '20 at 18:45
1
$\begingroup$

There is a possible ambiguity in "fit the data". Suppose first that a sample set of data points lie exactly on a straight line, so that the line estimated by linear regression with linear functional form provides a perfect fit (all residuals are zero). Suppose you take a sub-sample (of at least 2 non-coincident data points) and run the regression again. That will yield precisely the same regression line. This is true, moreover, whether the sub-sample is selected randomly from the original sample or whether it is selected so as deliberately to exclude data points associated with certain characteristics.

In an empirical study the above scenario is of course unlikely. Typically, there is a degree of randomness in the distribution of data points. Therefore, even if a line estimated by linear regression with linear functional form provides a good fit, the fit will be less than perfect (there will be some non-zero residuals). Because of the randomness, running the regression again with a sub-sample of the original sample probably will not yield precisely the same regression line.

If the original sample was large and a fairly large sub-sample is randomly chosen, then the likelihood is that the two regression lines will not differ very much. If however the sub-sample is deliberately chosen (eg as in your example to exclude married individuals) then it is possible that the two regression lines will differ considerably, because there may be on average a real difference between included and excluded individuals in the relationship between the regression variables. However, even if the two regression lines differ considerably, this does not imply that the linear functional form of the original line is inappropriate. It merely shows that the sample to which the original line was fitted includes sub-samples which differ in relevant ways.

$\endgroup$
3
  • $\begingroup$ Thanks for the answer - is there a point at which it would be more appropriate to present the OLSs over the subsamples rather than the OLS over the whole dataset (i.e. does the full-sample OLS relatively lose meaning at some point)? $\endgroup$ Nov 7 '20 at 22:46
  • $\begingroup$ @LawrenceWu I'm not sure I understand exactly what you are asking in your comment, but the following may help. Whenever you run a regression to estimate parameters, it is important to be clear as to what is the population of interest. If you have data for a random sample from your population of interest, there is nothing to be gained by running a regression on a sub-sample rather than on the whole sample (it would just tend to increase the standard errors of the parameter estimates). Running a regression on a sub-sample would be worthwhile in two circumstances: a) if you are interested ... $\endgroup$ Nov 8 '20 at 13:21
  • $\begingroup$ ... in estimating the parameters for a sub-population as well as for the whole population; b) if the population from which the original sample was drawn is not of interest to you (eg because you are using a dataset collected for another purpose), but a subset of that population is of interest to you. $\endgroup$ Nov 8 '20 at 13:24
0
$\begingroup$

The relationship may be different on a sub-sample if you purposefully exclude some demographics. The linear relationship between the education of males and wages and education of females and wages might not be the same. This is why wage equations control for the effect of gender. If you exclude gender the relationship might change even if the relationship is still linear for both males and females.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.