# Why can't we control for squares and quadratics of predicted values in regression?

If my goal were prediction (e.g. of propensity scores), why couldn't I control for higher order terms of the model equation? For example, why not estimate the model and then control for squares and cubes of the predicted values in a second stage and get an even better estimate of the correct predicted values (rather than just a test of functional form)? I know the standard errors would be wrong but couldn't I bootstrap the process?

• Why stop at cubes? Shouldn't you control for all powers all the way to the 99th? – Giskard May 25 '18 at 16:06
• There is a difference between using a flexible functional form and overfitting. With regards to my question, let's assume I'm not overfitting the model. – Kris B. May 25 '18 at 17:06
• Hi: Since, a regression is just minimizing a sum of squares between the value of the function used ( whatever functional form you like ) and the response, you can use whatever functional form you like but it doesn't mean the results are useful. I'm not clear on what you mean by "control for squares and cubes of predicted values" ???? – mark leeds May 25 '18 at 19:16
• Who said you can't? – Alecos Papadopoulos May 25 '18 at 19:57
• You can. The crux here is how you justify controlling for $x^2$ but not $x^3$, $x^4$, ...,$x^N$. You get into really arbitrary waters. Further, doing stuff like this will likely give you better in-sample fit but terrible out-of-sample predictions. It is an example of overfitting. So...you can...but within reason and with good justification. – 123 May 27 '18 at 15:50

For prediction, yes you can consider the models $$y = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p + \gamma_2 \hat{y}^2 + \cdots + \gamma_m \hat{y}^m + error,$$ where $\hat{y}$ represents the first-step OLS fitted values and $m$ is chosen by something like cross validation.
I haven't seen this approach used before. My personal guess is that this approach is not as useful as other commonly used ones (SVM, splines, GAM, etc.). For example, if $p$ is large (in comparison to the number of observations $n$), the first-step OLS may already be overfitting so including $\hat{y}$ is not practical. (Yes, you can use lasso residuals but that's a different story.) If $p$ is small, nonlinearity can perhaps be better handled by splines or even by simply augmenting the equation with quadratic and cubic terms of the features. Some generalized additive models (GAM) are already there too.