1
$\begingroup$

Im currently doing a project to construct a demand function for Meat & Fish. My data indicates that Price of Fruit & Veg is individually statistically significant. However, after a omitting this variable (as well as 3 other insignificant variables) the F-test result highlighted the failure to reject the Null (these four variables are jointly insignificant).

Should I therefore omit Fruit & Veg variable, despite it being individually significant and also making intuitive sense for inclusion into my demand function.

$\endgroup$
  • $\begingroup$ I find your question unclear. What test are you performing on which model variation? $\endgroup$ – Giskard Apr 17 '18 at 15:31
2
$\begingroup$

It is important to recognize that there are path dependencies to removing different variables from your model. This is due to the fact that variables can be highly correlated (positively or negatively) with one another so that by removing an insignificant variable then other variables can become statistically significant or insignificant.

There are very different approaches towards dealing with this. As the previous commenter mentioned you can use some sort of model selection procedure to try and explore the space of the variables you have included. There are many different approaches to doing this (AIC, BIC, SIC, Lasso, Gets...), but ultimately interpretation of the selected model assumes that your original model encompassed the true data generating process. If this is not the case, then selecting from your original model could give you inconsistent parameter estimates.

Alternatively you can stick with a purely theoretical approach and specify what you think the important variables are ahead of time (i.e. what you think the DGP is) and then only include those in your model. That way you don't add or remove any variables and you just interpret the estimated model based on your theory of what the true model is. Interpretation also relies on the assumption that your proposed model is in line with the underlying DGP.

$\endgroup$
2
$\begingroup$

You should do none of the above. This is an invalid decision-making process.

In the best of all possible worlds, create a series of do loops and go through the set of all possible combinations of variables. Calculate the AIC or the BIC. If you know nothing about either, just pick one as they usually give the same result.

The model with the lowest AIC or BIC is the model that is probably the closest to the true model in nature. Statistical significance does not matter at all. If the best model is also not significant, then it is not significant. It could just mean the other models have a spurious significance. Likewise, if the model chosen is significant under the F-test, but has non-significant variables, then you cannot change them.

If you do not know how to write do loops or for-next loops, then find a package with step-wise regression. It will not cover all the models, but it will cover many. Use the same criterion either the AIC or the BIC.

Because you are asserting the null is true, you cannot have more than one null. It means nothing if you add or subtract variables because you are changing the null every time and you cannot do that. The AIC or BIC are non-Frequentist methods of estimating the true model and so the solution of using either bypasses the question of significance until after the model is chosen.

EDIT

I thought I would provide an edit to cover the statements in the comment.

For starters, I agree with the comment. I thought I should ground the above statement better as to the logic behind it.

The various information criteria, the AIC, the BIC, the DIC and so forth, can either be grounded in information theory or in Bayesian theory.

From an information theoretic perspective, if you have outside information about which models should be included or excluded either from theory or experience then that information needs to be incorporated. Since there is no direct method to merge them together, you should use judgment in which models to look at.

From a Bayesian perspective, the various information criteria are stylized point approximations of the Bayesian posterior under somewhat restrictive assumptions. In many respects, they are not good proxies because the smaller the posterior density, the less likely it is to be true, whereas the reverse is true in the criterion. It is best to think of them as both being rankings and they would provide the same ordering of rankings.

Now there are two Bayesian issues present. First comes from the fact that you can construct Bayesian theory from Cox's axioms. Cox's axioms are built around Aristotelian logic. You would use Bayesian methods to assess logical statements. If some statements do not need assessing or could be excluded by logic, then they should be excluded from consideration.

The second comes from the nature of the prior density. If you have prior knowledge that some cases cannot be true, then you should give them zero prior weight. This would exclude them from consideration at all.

Nonetheless, the combinatoric method should be considered because it maps to a Bayesian argument and the information criterion are non-Frequentist constructions and so you shouldn't be using Frequentist criteria in making this type of decision.

Bayesian hypotheses are combinatoric. If you are testing the model $$y=\beta_0+\beta_1x_1+\beta_2x_2+\beta_3x_3+\varepsilon$$ and you have no logical reason to exclude any case, then you are going to have to test eight hypotheses, removing one, two or three variables at a time. This is the practical equivalent to the Frequentist hypothesis that $\beta_i=0$ which cannot be tested in Bayesian methods because it is a sharp null hypothesis and of measure zero. Removing a variable and assuming its coefficient is zero is the same thing.

Because Bayesian methods lack a null hypothesis, which is both a strength and a shortcoming, no one hypothesis gets special treatment or special weight, absent material external information. You test the posterior probability of each possible model as part of your process.

Now, this is where a legitimate objection does come in. Information criteria are stylized approximations of the actual posterior. In some cases, they are perfect approximations but they can be poor approximations in other cases. In the perfect approximation case, you should be using the information criterion that is appropriate for your problem and your worry is not about the criterion, but rather the representativeness of your data. In the poor approximation case, close differences could, in fact, be reversed if the posterior was actually computed. As you multiply your combinations, the greater the probability of a single pairwise mistake will happen. Nonetheless, since you are not doing model averaging, but model selection it will probably not be the highest two pairs.

Using tools like the information criterion does highlight a problem in data-based measures over a true null hypothesis based methods, the combinations show how little independent data that you may have. If you are not model hunting, but instead testing the one true model, then you will lose a few degrees of freedom and unless you have collinearity problems, then you are fine if your sample is of reasonable size. In my example above, it is quite like dividing your sample by eight. It isn't a subtractive process as one would see with degrees of freedom, it is more like a process of division. Add into that the internal correlations and there may not be much in the way of independent information in your set.

Still, it didn't seem to me like you felt you had a mental model of how the relationships should be, so I would still recommend either the combinatoric or the step-wise solution.

$\endgroup$
  • $\begingroup$ Ok, so perhaps I am too much trained in the old frequentists' methods, but this is a bit too data-driven approach for my taste. I'm fine with comparing a few models based where the decision to include or exclude variables are based on theory and possibly parsimony using the AIC (or BIC), but running through all possible combinations looking for the lowest AIC seems like a fishing expedition (see: stats.stackexchange.com/questions/78949/… ). In that way one could include anything including your birthday $\endgroup$ – Maarten Punt Apr 19 '18 at 7:24
  • $\begingroup$ @MaartenPunt I agree with you and provided an edit. I am a giant opponent of fishing expeditions, but the OP's question didn't have the appearance of fishing in this case. Rather, it was confusion. $\endgroup$ – Dave Harris Apr 19 '18 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.