2
$\begingroup$

Imagine we have a linear regression model with dependent variable $y$. We find its $R^2_y$. Now, we do another regression, but this time on $\log(y)$, and similarly find its $R^2_{\log(y)}$. Why can't I compare both $R^2$ to see which model is better suited?

Intuitively I would say that the log will decrease the variability, hence any model will improve once we 'add' the log transformation.

Is there another reason? Also is there a way to formalise this intuition?

Any help would be appreciated.

$\endgroup$
  • $\begingroup$ You are only transforming the dependent variable and keep the regressors the same? Or the question is more general "Can we use $R^2$ as a model selection criterion"? $\endgroup$ – Alecos Papadopoulos May 3 '17 at 21:29
  • $\begingroup$ Unless you have information not shown here, there is not particular reason to suppose $\log$ will improve the $R^2$. It might, or it might have the opposite effect $\endgroup$ – Henry May 3 '17 at 21:32
  • $\begingroup$ @AlecosPapadopoulos I'm interested in both cases, while keeping the same number of regressors. $\endgroup$ – An old man in the sea. May 4 '17 at 8:08
  • $\begingroup$ @Henry at first thought, I would tend to disagree if the values are big enough(maybe 10^3 or more?). The log transformation is known precisely by its variance reduction properties. $\endgroup$ – An old man in the sea. May 4 '17 at 8:11
  • $\begingroup$ Suppose your $x$ data is $1000, 2000, 3000$ and your $y$ data is $4000, 6000, 8000$. Taking logarithms will reduce $R^2$ $\endgroup$ – Henry May 4 '17 at 8:21
1
$\begingroup$

$R^2$ measures the fit of the model to the data. Notice the ordering of that sentence, "model to the data," and not "the data to the model." You want to use $R^2$ as a criterion to select among two possible equations. The fact that you are posting here also probably means either someone told you that you cannot do that or you read online that this will not work. The fact is, it can work, but it does not function well.

Several aspects of how $R^2$ is calculated make it an inadequate criterion. First, and this is the classical reason, all you have to do to increase $R^2$ is add variables. If you were predicting the impact of milk substitutes on babies' performance on a physical test and you added in the price of flawless, color D, three carat diamonds as traded in Hong Kong your $R^2$ may go up, but it will not go down. In addition, your total sum of the squares will change under the transformation you propose. Because of this, it is not well suited to using an F-test to compare models.

This brings you to the AIC and the BIC. Philosophically, the AIC and the BIC are algorithmic approximations of the Bayes factor to choose between the two models. Because you know your data and we do not, you should read a book on model selection, such as this one.

However, given the limited information that you did provide, it would appear, at least on the surface, that either the AIC or the BIC would produce equivalent results. That is because you have only two models and they have an equal number of parameters.

If you have not used a Bayesian method before, they reverse the direction of probability. Instead of assuming a model is true and determining whether or not the data is as extreme or more extreme than some standard, it assumes the data is fixed and not random and that the models are uncertain, and so selects the best parameters and models based on the data. There is no null hypothesis. So the AIC or BIC are approximations of odds transformed into an algorithmic rule.

They differ in two ways. First, the BIC gives each model equal prior probability of being "true," while the AIC gives probabilities in proportion to their number of parameters so that complex models are penalized for being complex. Second, they approximate the likelihood function differently so that the AIC ends up penalizing complex models less than the BIC unless the sample size is large.

The reason to use either tool, rather than a full-blown Bayesian method, is that they are faster, they are good approximations in most circumstances to the Bayesian solution, and they are less complex than a Bayesian model selection process.

You cannot formalize the idea of the logarithm reducing volatility and creating nicer models because it may not do so. Consider a model whose true form is $y=5x+7$ and you then transform those variables. A log-linear model would be expected to be a worse model than the untransformed model. Although using logs would reduce the scale of all variables, $R^2$ is based on the relative scale. Dividing all variables by two would also reduce variability, but not improve $R^2$ or model selection.

$\endgroup$
0
$\begingroup$

In order to think about the intuition it is important to understand how $R^2$ is computed. We can think of the R^2 as the ratio of the estimated variation over the total actual variation of the dependent variable: $R^2=\frac{SSE}{SST}$ or $R^2=1-\frac{SSR}{SST}$ where $SSE$ is the estimated sum of squares, $SSR$ is the residual sum of squares and $SST$ is the total sum of squares. So what the $R^2$ tells us is how much of the variation in the dependent variable that our model captures. However, whenever we change the dependent variable we not only change $SSE$ or $SSR$ but we also change $SST$. This implies that the new $R^2$ is now using a different base of comparison. Thus, it is generally not comparable for different dependent variables due to the fact that SST is different. Does this make sense?

In time series models this is a fairly big concern since modelers often play with the dependent variable by differencing which impacts the $R^2$ and make comparisons invalid even when in some cases it has no actual impact on the residuals. As a result, it is common to use residual based metrics that are invariant to these types of transformations.

$\endgroup$
  • 1
    $\begingroup$ I believe that the OP asked his question in light of the fact that $R^2$ appears to be a relative measure, so, at least apparently, free of units-of-measurement / base of calculation. It is exactly because it not only changes $SSR$ but also $SST$ that creates in principle the possibility of using it also as a model comparison and selection criterion. $\endgroup$ – Alecos Papadopoulos May 4 '17 at 7:12
  • 1
    $\begingroup$ But by changing $SST$ implies that $R^2$ is no longer base free and it is therefore useless to compare between the two measures. Yes you can understand how changes in the independent variables impact the model, but not how changes in the dependent variable do. It sounds like what you are looking for is a rule that guides the link between these changes in base. $\endgroup$ – Andrew M May 4 '17 at 9:52
  • $\begingroup$ @AndrewM is precisely «changing SST implies that R2 is no longer base free and it is therefore useless to compare between the two measures.» this conclusion that I would like to see formalized in a reasoning. Can you do it? Thanks for you answer. ;) $\endgroup$ – An old man in the sea. May 4 '17 at 18:05
0
$\begingroup$

The simple answer is that we're comparing different dependent variables, and usually these imply different models, some of which do not satisfy the usual hypotheses. In this case, we have

  1. Model with dependent variable $y_t$ implies that $\log(y_t)$ is not linear in the parameters.
  2. Model with dependent variable $\log(y_t)$ implies that $y_t$ is not linear in the parameters.

So, only one can satisfy linearity. There's no point in comparing models. We only compare them, after we've ascertained that they comply with the hypotheses.

We can just check which one obeys linearity from a data plot, or other technique, and then choose that one.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.