Imagine we have a linear regression model with dependent variable $y$. We find its $R^2_y$. Now, we do another regression, but this time on $\log(y)$, and similarly find its $R^2_{\log(y)}$. Why can't I compare both $R^2$ to see which model is better suited?
Intuitively I would say that the log will decrease the variability, hence any model will improve once we 'add' the log transformation.
Is there another reason? Also is there a way to formalise this intuition?
Any help would be appreciated.
$R^2$ measures the fit of the model to the data. Notice the ordering of that sentence, "model to the data," and not "the data to the model." You want to use $R^2$ as a criterion to select among two possible equations. The fact that you are posting here also probably means either someone told you that you cannot do that or you read online that this will not work. The fact is, it can work, but it does not function well.
Several aspects of how $R^2$ is calculated make it an inadequate criterion. First, and this is the classical reason, all you have to do to increase $R^2$ is add variables. If you were predicting the impact of milk substitutes on babies' performance on a physical test and you added in the price of flawless, color D, three carat diamonds as traded in Hong Kong your $R^2$ may go up, but it will not go down. In addition, your total sum of the squares will change under the transformation you propose. Because of this, it is not well suited to using an F-test to compare models.
This brings you to the AIC and the BIC. Philosophically, the AIC and the BIC are algorithmic approximations of the Bayes factor to choose between the two models. Because you know your data and we do not, you should read a book on model selection, such as this one.
However, given the limited information that you did provide, it would appear, at least on the surface, that either the AIC or the BIC would produce equivalent results. That is because you have only two models and they have an equal number of parameters.
If you have not used a Bayesian method before, they reverse the direction of probability. Instead of assuming a model is true and determining whether or not the data is as extreme or more extreme than some standard, it assumes the data is fixed and not random and that the models are uncertain, and so selects the best parameters and models based on the data. There is no null hypothesis. So the AIC or BIC are approximations of odds transformed into an algorithmic rule.
They differ in two ways. First, the BIC gives each model equal prior probability of being "true," while the AIC gives probabilities in proportion to their number of parameters so that complex models are penalized for being complex. Second, they approximate the likelihood function differently so that the AIC ends up penalizing complex models less than the BIC unless the sample size is large.
The reason to use either tool, rather than a full-blown Bayesian method, is that they are faster, they are good approximations in most circumstances to the Bayesian solution, and they are less complex than a Bayesian model selection process.
You cannot formalize the idea of the logarithm reducing volatility and creating nicer models because it may not do so. Consider a model whose true form is $y=5x+7$ and you then transform those variables. A log-linear model would be expected to be a worse model than the untransformed model. Although using logs would reduce the scale of all variables, $R^2$ is based on the relative scale. Dividing all variables by two would also reduce variability, but not improve $R^2$ or model selection.
In order to think about the intuition it is important to understand how $R^2$ is computed. We can think of the R^2 as the ratio of the estimated variation over the total actual variation of the dependent variable: $R^2=\frac{SSE}{SST}$ or $R^2=1-\frac{SSR}{SST}$ where $SSE$ is the estimated sum of squares, $SSR$ is the residual sum of squares and $SST$ is the total sum of squares. So what the $R^2$ tells us is how much of the variation in the dependent variable that our model captures. However, whenever we change the dependent variable we not only change $SSE$ or $SSR$ but we also change $SST$. This implies that the new $R^2$ is now using a different base of comparison. Thus, it is generally not comparable for different dependent variables due to the fact that SST is different. Does this make sense?
In time series models this is a fairly big concern since modelers often play with the dependent variable by differencing which impacts the $R^2$ and make comparisons invalid even when in some cases it has no actual impact on the residuals. As a result, it is common to use residual based metrics that are invariant to these types of transformations.
The simple answer is that we're comparing different dependent variables, and usually these imply different models, some of which do not satisfy the usual hypotheses. In this case, we have
So, only one can satisfy linearity. There's no point in comparing models. We only compare them, after we've ascertained that they comply with the hypotheses.
We can just check which one obeys linearity from a data plot, or other technique, and then choose that one.
