(1) why is imposing CES so important in our models?
Because although its relatively quite general (relatively to some other widely used production functions like Cobb-Douglas - which is a special case of CES) it is still easy to estimate with parametric models and generally CES production functions are easy to work with (McFadden 1963).
Until very recently you need Cobb-Douglas, or some CES, with its unitary elasticity of substitution due to the normalization problem which precluded people from applying even more general form. For example, as discussed in Klump et al (2011) [emphasis mine]:
Until recently, the application of production functions
with non-unitary substitution elasticities (i.e., non Cobb Douglas) was hampered
by empirical and theoretical uncertainties. As has recently been revealed, “normalization” of production functions and production-technology systems holds out
the promise of resolving many of those uncertainties and allowing considerations as
the role of the substitution elasticity and biased technical change to play a deeper
role in growth and business-cycle analysis. Normalization essentially implies representing the production function in consistent indexed number form. Without
normalization, it can be shown that the production function parameters have no
economic interpretation since they are dependent on the normalization point and
the elasticity of substitution itself. This feature significantly undermines estimation and comparative-static exercises, among other things.
The above issue leads to bias in estimation (especially in parametric models) so its quite a serious issue. This is because, we can only use observable data for estimation but capital and labor are measured in completely different units (aside from the problem that we actually have no way of accurately measuring capital in the first place). CES or Cobb-Douglass with unitary fixed elasticity gets around the issue by the virtue of elasticity of substitution being 1, and because the differences in units get absorbed into the scaling constant.
But what even more, as the above cited paper discussed, the problem of normalization was essentially solved for CES even with non-unitary elasticities making it even so more desirable function to use. This is quite important since empirically elasticity of substitution is generally below unity (e.g. have a look at Chirinko et al. 1999, Klump et al. 2007, Leon-Ledesma et al. 2010).
Lastly, estimation of production functions is riddled beyond belief with endogeneity issues. One way how to solve the endogeneity issues is to look to theory for guidance of how to set up model (especially how to properly specify the error term) to avoid these issues (e.g. see the work of Olley Pakes 1996, Levinsohn-Petrin 2000 or Ackerberg, Caves and Frazer 2015). But lot of these theoretical results are derived assuming CES production function, so you can't just use theoretical result derived assuming CES production to model the structure of the error term and nothing else.
- are there any serious papers or methods that allow for a non-CES production function?
Well yes, there are a lot of papers that apply translog production function which allows elasticity of substitution to change (Heathfield, Wibe 1987). If you just put translog production function estimation into google scholar you will get a lot of examples.
However, estimating translog production function parametrically can be problematic because relative to CES or Cobb-Douglass you have to estimate quite a lot of parameters to make it work. Especially if you want to include a lot of factors of production the number of parameters virtually explodes (Pavelescu, 2011), and this is a problem given the tendency of including more and more factors in recent years (e.g. materials, different types of capital and so on).
