Which measure to check in analysis (p-values or $R^2$)?

Question

In an analysis e.g. OLS regression which measure would you look at?

$R^2$, $adjusted-R^2$ or $p-value$?

How would you consider a regression with p-values >0.15 but with an $R^2$ of 40%? In general in economics, what have you seen?

Answer 1

R-squareds and p-values give different information. For example, suppose that you regress birth weight on mother's smoking. The p-value tells us whether the association is indeed nonzero (in the statistical sense), and the R-squared tells us how much of variation in birth weight is explained by mother's smoking.

p-value > .15 and R-squared 40% would mean that (1) we can't say that the association is nonzero (2) but the explanatory power for the given sample is quite large. The two neither reinforce each other nor are contradictory.

Answer 2

If you are running a regression analysis I would say you should look first at p-values to check if the explanatory variables you added make sense. I mean if you have a hypothesis about a relationship among two variables and you find the explanatory one not significant (large p-value) I would check for possible mistakes or alternative explanations.

It makes sense to have a look at $R^2$ if you are comparing two different specification, it does not really make sense looking at it alone. I have seen many papers, have a look at Nunn, Wantcheckon, (2011) as an example, which have been published on top journals (AER in this case) even with $R^2$ around 16%, not really an high value.

Answer 3

There is in fact a theoretical answer to this and it may be of importance to you in the long run. There are three main schools of statistical thought and several minor schools. The three are, in the order of discovery, the Bayesian, the Likelihoodist and the Frequentist. The differences matter because they can provide different answers using the exact same data and running the same formula.

From the language of your posting, you are not using the Bayesian methodology. Bayesian methods do not have a null hypothesis, so they cannot have a p-value. Rather than compute a probability that the null is as extreme or more extreme than you would expect if the null is true, Bayesian methods compute the positive probability that each separate hypothesis is true. It does not falsify a null, it assigns a positive probability to each competing possible hypothesis. It does not limit itself to two hypothesis, though you can have two.

The second is the Likelihoodist model and this is where the language in your posting seems to go. The likelihoodist model has p-values, but no cut-off value, $\alpha$, while the Frequentist school as a cut-off value $\alpha$, but there is no such thing as a p-value. Both schools are part of a broader scheme called the null hypothesis method.

In the null hypothesis method, you choose a null and grant 100% probability that it is true, ex ante. Statistics is a branch of rhetoric and not a branch of mathematics. Like physics, it uses mathematics intensively, but it is not part of the field of mathematics. Rather, it is about how to decide arguments. The ordinary null used in statistics is the "no effect" hypothesis. That is you argue that all $\beta=0$, that is the independent variables have no effect at all on the dependent variable. If you falsify it, then you are arguing that it does have an effect and the null is falsified.

In the case of the Likelihoodist school, there is no magical cut-off value for the p-value. You just report it. It is the weight of the evidence against the null. A p-value of less than .15 leaves a lot of room for chance, but if you are okay with that level, then it is significant to you. This is an either-or discussion. It either is good enough, or it is not good enough.

If you believe it is good enough, then you use the values of the OLS equation because they are the most probable value based on your data. From that point onward, the adjusted $R^2$ gives the best measure of the amount of variability that your model provides.

If you do not believe that a p-value is good enough, then you are faced with a peculiar problem. You need to throw away your OLS equation, according to theory. If the null is not falsified then no new information exists and you ignore your results. It is as if you had never performed the analysis in the first place. The Likelihoodist school is epistemological, that is it is a knowledge seeking tool. If the p-value is not adequate for your purposes, then there is no added knowledge and you need to spend your life looking at something else rather than waste your time on this topic.

On the other hand, if you use the Frequentist school, the results have a different interpretation. In the Frequentist school, the OLS equations are not the maximum likelihood estimator(MLE) as they are with Likelihoodists, but instead the equations are the minimum variance unbiased estimators(MVUE). It happens to work out nicely that the two schools have the same answers for the simple problems, such as regression.

The Frequentist school is behavioral. It tells you how to behave. For example, if you were doing quality assurance on batches coming out of a factory and you got results indicating poor quality, falsifying the null of good quality, then you destroy the batch, otherwise you accept the batch. You do not know the batch is bad just because it tests bad. You do not know if a batch is good, just because it tests good. This is not about knowledge, it is about how you should behave. Do you destroy the batch when you reject the null and accept the batch when you accept the null? Yes.

In order to create this, before you collect the data, you set a value called $\alpha$, which is your cut-off. This value is set with respect to both the importance of false positives and the importance of false negatives. If the test statistic is in the rejection region, then you behave as if the OLS parameters were the true parameters. The adjusted $R^2$ then provides the degree of variability explained by the model. If the test statistic is in the acceptance region, then you have accepted the null. The null was that all $\beta=0$, so you are obligated to behave as if there was no relationship between any of the independent variables and the dependent variable and the $R^2$ doesn't matter because it is the the $R^2$ of a model that you are not using.

It is difficult to determine what is going on with your model using null hypothesis methods. It could be spurious correlations that explain a lot of the variability of your sample, but not the variability that you would find out of sample. It could be that the effects are close to zero, but not zero, but the effect size is too small for it to be clearly significant. That is to say, your sample lacks the power to detect the effect. The problem with null hypothesis methods, in general, is that there is no way to distinguish a null that is truly false from one where the result was due to a weird sample. It is impossible to distinguish truth and chance effects.

If you have a professor around that uses Bayesian methods, and they are uncommon in economics, you could have them run it as a Bayesian model. Bayesian hypotheses are combinatoric, so that if you have three regressors, $x_1, x_2, x_3$, the researcher has to run all possible combinations of dependent variables and test each submodel for the probability that it is the true model. This may help distinguish between the various parts and the things that can go wrong with MLE and MVUE models such as multicolinearity, small effects and so forth.

Answer 4

Well, adjusted R2 has more merit than R2. R2 will always rise (or at least stay the same) as you add more variables (even if the variables are not statistically significant). Thus in theory you could achieve a very high R2 by just bunging a load of nonsense variables into your regression. Adjusted R2 compensates for this by adjusting the R2 by a penalization factor that increases as the number of variables rise - hence adding a new variable will always raise r2 but will raise adjusted r2 only based on its explanatory power. Information criteria such as the Bayesian information criterion (also called Schwarz criterion) or Akike information criterion can be used to choose between competing models. Intuitively, they measure explanatory power but penalize for "bigger" models which are more prone to overfitting and other problems. The BIC penalizes additional parameters more heavilly than the AIC, for example. You would prefer the model which minimizes the information criterion score. Note that, unlike R2, the values of information criteria are not inherently very meaningful - you cannot say "this is a good model because it has a BIC of -50", but you can use them to choose between.

What matters more than either P values or measures of goodness of fit is that the model is theoretically sensible and that the Gauss-Markov assumptions hold. This is what makes a "good" model - ability to explain a high proportion of the variance is the icing on the cake, nothing more. If I regress rainfall in the UK on US GDP for instance, I will get a P-value of close to zero and an R2 of above 0.95 - yet obviously there is no genuine economic relationship here.

So a sensible model with solid theoretical foundations which satisfies the OLS assumptions is a good model even if its explanatory power is limited. Conversely a model with high r2 / low p values which doesn't satisfy econometric assumptions or lacks theoretical credibility can never be a "good" model.

Answer 5

If you want to forecast it is common practice to split your sample up into two parts - the "estimation sample" and the "forecast sample". The estimation sample is used to, you guessed it, estimate your model, which you then use to forecast the data for which you already have true values contained in your forecast sample. You then compare your forecast predictions to known data and to those of other forecasts.

You can do this using r2 or adjusted r2 over the forecast sample. We also commonly use measures like means squared error, mean absolute percentage error, Theil's U, etc. etc.