I have completed an F-test on the variable which gives p = 0.6, should I remove it from the model?
3
-
$\begingroup$ Yes you can. If it is just a simple dummy variable for a single observation then it is orthogonal to other variables in the model and is just eating up one degree of freedom. So unless there is a theoretical reason for including I would remove. $\endgroup$– Andrew MApr 29, 2020 at 13:50
-
1$\begingroup$ Can you present the model? It would be interesting to know what the variable is. Also what F-test deals with a single variable, there is more going on than a simple test of significance. Please provide your results! $\endgroup$– BrennanApr 29, 2020 at 18:31
-
$\begingroup$ If you are interested in causal relationship, pretesting is dangerous. $\endgroup$– chan1142Apr 30, 2020 at 0:37
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Statisticians object to this kind of pre-testing. This is a form of stepwise regression, a methodology known to be problematic.
Two issues particularly bother me, both of which are discussed further in the links.
If you do any kind of testing (keep in mind that confidence intervals are just inverses of tests), the p-values are lying, since you also should condition on having performed a pre-test (but the usual p-values do not do this). This corresponds to criticisms 2 and 4 in the Harrell article I link at the end of this post.
Even if you just want to predict, your measures of performance are likely to be inflated. In particular, $R^2_{adj}$ and the usual “unbiased” estimator of error term variance will use the degrees of freedom from the model fitted after the pre-test. However, you did consider that additional variable, and it should count towards the degrees of freedom. This corresponds to criticism #1 in the Harrell article I link at the end of this post.
Frank Harrell and Andrew Gelman have written some thoughts on stepwise regression that expand on my comments here.
