My data has issues of multicollinearity. Upon estimating, I found out that my data have high levels of VIF In however I was confused as to what level of VIF can I possibly ignore the issue of multicollinearity.

As I search the web, some says that it should be less than 2.5, others 10 or 14.

My VIF results is this:

  • varA: 18.74
  • varB: 11.14
  • varC: 10.03
  • varD: 5.99
  • varE: 2.86

What step should I probably do to address the issue? I have fixed sample size and I can't log transform my model since one of my variables ranges from a negative to positive number.

  • $\begingroup$ Are you interested in the interpretation of coefficient estimates and p-values for all of these variables? If multicollinearity is only present among control variables, whose interpretation does not concern you, then you may not need to fix anything. $\endgroup$ – AlexK Apr 9 '19 at 21:20

There are two potential ways out of this problem. The first is to perform principal components analysis (PCA) to reduce your variables in use. The second is to construct it as a Bayesian problem.

The difficulty with PCA is two-fold. First, the results are very difficult to interpret and explain. Second, if there is a theoretical reason for certain variables to be linked and you eliminate that link in the PCA, then regardless of how good your model works out to be, it's wrong.

The Bayesian alternative is a great alternative if you have at least weak information of the relationships from information outside the sample itself, such as other studies. Multicollinearity is a non-problem for Bayesian methods. The only issue that would ever form would be with perfect multicollinearity, in which case you really only have one variable and you need to pick one to use.

It is because of several factors. First, Bayesian methods are intrinsically optimal methods, ex-post. All Bayesian estimators with proper priors are admissible, Frequentist estimators are admissible only to the extent they match a Bayesian estimator at every sample or match a Bayesian estimator at the limit. This optimality, in part, comes from the fact that the likelihood function is always minimally sufficient for the parameters. There is no way to get more information out of a data set and there is no less risky way to calculate an estimate. Bayesian estimators cannot be stochastically dominated.

They also ignore redundant information. This isn't from a duplicate observation, rather if information has already been encoded (not noise) into a calculation then identical information doesn't get encoded. If you have variables that covary highly, there is little independent information in the sample. Most of the information is shared among the variables. In Bayesian calculations, the shared information gets added one time. Frequentist methods can have problems with multicollinearity because that isn't necessarily true for them. That is the reason for the assumption of no linear dependence.

The problem with Bayesian methods is that you have to encode all outside knowledge about the relationships from whatever appears in the literature into a prior distribution. The prior distribution encodes all the information you have about the relationships that exist outside the sample itself. The prior is then multiplied by the likelihood of seeing the sample and is averaged over the parameter space.

The nice thing about Bayesian methods is that they are rarely less accurate than a Frequentist method and often more accurate, sometimes by orders of magnitude. There was an example problem in the 1930s by R.A. Fisher and his criticism of the minimum variance unbiased estimator was that it could be accurate, in the sense that the errors were symmetric, but incorrect where correctness would be what we now call precision. OLS can be very far from the true value when compared to Bayesian methods. With large sample sizes, however, they tend to converge both in parameter estimates and accuracy.

If you have never performed Bayesian calculations, you will want to get professional help first. The same is true of principal components analysis.

In the case of a Bayesian method, data isn't considered a random variable. There is no such thing as a random sample. Data is fixed. You saw it. There is nothing uncertain about it. The parameters are considered random in the sense that there is uncertainty about them.

A Bayesian can say "its probably raining," and it makes sense. To a Frequentist, that is nonsensical. It is either raining or it is not raining. It is either 100% true or 100% false as it is a fact.

For a Frequentist, it makes sense to reject a null when $\bar{x}>k$ if being outside a range would unusual if the null is true in the sense of it being unlikely to be as extreme or more extreme. The Bayesian might reject that rejection. The Bayesian may ask, "has anyone actually observed a result bigger than $\bar{x}$? How do you know a more extreme result could even exist? How can you make inferential judgments on something that has never been seen?

The weird element to principal components analysis is that it implies a geometry. You are going to rotate reality until reality splits apart into orthogonal components. You could also use factor analysis if you believed there were latent factors in the data. In either case, you won't be talking about the world as you know it. Instead, PCA will attempt to sieve out the hidden parts into new components that have no social meaning. It becomes just math.

| improve this answer | |
  • $\begingroup$ Thank you for your explanation. I will further look into it since it was the first time I encountered such methods. $\endgroup$ – Lael2117 Apr 9 '19 at 1:53

I cannot post this as a comment but researchers such as Farrar and Glauber (1976) have shown that if VIF ≥ 10, this indicates the presence of multicollinearity between explanatory variables. There is no black and white threshold for this within the literature, and it highly depends on your sources and on the person reviewing your work. Moreover, you have not made any mention of the estimator you are using, but as you may know, multicollinearity makes the OLS estimator is less precise. That being said, have you tried standardizing by subtracting the mean? More information on your model and or your own knowledge on this would be useful to get help, in my view.

| improve this answer | |
  • $\begingroup$ Thank you, I use OLS but what do you mean by standardizing the mean? In addition, I have 16 ob and in my study I was trying to found out if one variable have such an effect on inequality. $\endgroup$ – Lael2117 Apr 9 '19 at 1:56
  • $\begingroup$ @Lael2117 The method of subtracting the mean from continuous variables is recommended when you have "structural multicollinearity" (i.e., when you create a model term using other terms, like an interaction term). In this case, it is recommended to standardize/center the main terms via this method. More info can be found here: statisticsbyjim.com/regression/… $\endgroup$ – AlexK Apr 9 '19 at 21:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.