If my goal were prediction (e.g. of propensity scores), why couldn't I control for higher order terms of the model equation? For example, why not estimate the model and then control for squares and cubes of the predicted values in a second stage and get an even better estimate of the correct predicted values (rather than just a test of functional form)? I know the standard errors would be wrong but couldn't I bootstrap the process?

  • $\begingroup$ Why stop at cubes? Shouldn't you control for all powers all the way to the 99th? $\endgroup$
    – Giskard
    May 25 '18 at 16:06
  • $\begingroup$ There is a difference between using a flexible functional form and overfitting. With regards to my question, let's assume I'm not overfitting the model. $\endgroup$
    – Kris B.
    May 25 '18 at 17:06
  • $\begingroup$ Hi: Since, a regression is just minimizing a sum of squares between the value of the function used ( whatever functional form you like ) and the response, you can use whatever functional form you like but it doesn't mean the results are useful. I'm not clear on what you mean by "control for squares and cubes of predicted values" ???? $\endgroup$
    – mark leeds
    May 25 '18 at 19:16
  • 4
    $\begingroup$ Who said you can't? $\endgroup$ May 25 '18 at 19:57
  • 1
    $\begingroup$ You can. The crux here is how you justify controlling for $x^2$ but not $x^3$, $x^4$, ...,$x^N$. You get into really arbitrary waters. Further, doing stuff like this will likely give you better in-sample fit but terrible out-of-sample predictions. It is an example of overfitting. So...you can...but within reason and with good justification. $\endgroup$
    – 123
    May 27 '18 at 15:50

For prediction, yes you can consider the models $$ y = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p + \gamma_2 \hat{y}^2 + \cdots + \gamma_m \hat{y}^m + error, $$ where $\hat{y}$ represents the first-step OLS fitted values and $m$ is chosen by something like cross validation.

I haven't seen this approach used before. My personal guess is that this approach is not as useful as other commonly used ones (SVM, splines, GAM, etc.). For example, if $p$ is large (in comparison to the number of observations $n$), the first-step OLS may already be overfitting so including $\hat{y}$ is not practical. (Yes, you can use lasso residuals but that's a different story.) If $p$ is small, nonlinearity can perhaps be better handled by splines or even by simply augmenting the equation with quadratic and cubic terms of the features. Some generalized additive models (GAM) are already there too.

My personal experience is that nonlinearity is not so important (for prediction using economic data). It is usually a lot more important to avoid overfitting nicely. To me, your suggestion seems to be useful in some cases but not in many.

That said, I do not want to dissuade you from pursuing this issue, although there is a (high) chance of ending up with the conclusion that it is not very useful given the availability of other methods. BTW, you would already know it, but just in case, Hastie, Tibshirani and Friedman's book (The Elements of Statistical Learning) is helpful.

  • $\begingroup$ Hi: you may also want to look at discussions of the autoregressive distributed lag or the koyck distributed lag. These are not terribly re econometric approaches not terribly related to what you suggest and they won't talk about why your approach ( I think you're approach is to fit as many powers as possible which is prone to overfitting ) is not necessarily useful but you may find them interesting as approaches on their own. they are both time-series modelling approaches so, if you're problem is not time series related, they won't be helpful. $\endgroup$
    – mark leeds
    May 29 '18 at 2:42
  • $\begingroup$ @markleeds Regarding your concern of overfitting, the answer mentions that the degree of the polynomial (model selection) is performed via something like cross-validation. Cross-validation exists precisely to avoid overfitting. Most econometrics textbooks don't use cross validation, because their primary concern is inference rather than prediction. $\endgroup$
    – jmbejara
    Aug 21 '19 at 13:42
  • 1
    $\begingroup$ @jmbejara. good point about cross-validation. thanks. $\endgroup$
    – mark leeds
    Aug 21 '19 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.