I would recommend reading Thaler's Misbehaving, which chronicles the development of behavioral economics as a field and its struggle to gain recognition by mainstream economists. Several of its chapters (particularly, 6, 17-20) would directly answer your question regarding the historical context and partly the "why".
I share my own personal view below.
I believe the "as if" argument as advanced by Milton Friedman in his Essays in Positive Economics plays an important role in countering the move towards a relaxation of the rationality assumption. In a nutshell, the argument says that a theory should be judged by the accuracy of its predictions, not by the realism of its assumptions. Therefore, while the assumption of a rational decision-maker is not entirely realistic, the theory is doing a good job as long as it generates sufficiently accurate predictions (in the sense that they match what actually happens in the real world) based on such an assumption. Friedman describes a simple of model of billiard players that I think nicely illustrates his idea:
Consider the problem of predicting the shots made by an expert
billiard player. It seems not at all unreasonable that excellent
predictions would be yielded by the hypothesis that the billiard
player made his shots as if he knew the complicated mathematical
formulas that would give the optimum directions of travel, could
estimate accurately by eye the angles, etc., describing the location
of the balls, could make lightning calculations from the formulas, and
could then make the balls travel in the direction indicated by the
formulas. Our confidence in this hypothesis is not based on the belief
that billiard players, even expert ones, can or do go through the
process described; it derives rather from the belief that, unless in
some way or other they were capable of reaching essentially the same
result, they would not in fact be expert billiard players.
Thaler, and Kahneman and Tversky before him, mounted copious evidence that people consistently deviate from the behavioral benchmark set by the homo economicus. But to people like Friedman, this is not a sufficient cause for a change of paradigm, unless one brings conclusive evidence to show that the predictions made by rational choice models are wrong. It didn't help that experimental economists like Vernon Smith were successfully showing that (in the market context) people's collective behavior leads to outcomes consistent with rational choice theory. (In this regard, it does seem a bit ironic that Smith would share a Nobel with Kahneman).
Another reason against the paradigm shift could be in the interest of parsimony. Including more realistic behavioral foundations in economic models inevitably makes the models more complex. But if the simpler, albeit less realistically-founded, rational choice model can get things right just as well, there is a philosophical reason for favoring the latter. In a way, this is similar to the if-it-ain't-broke-don't-fix-it mentality.
Eventually, however, rational choice theory were found to fail to predict a lot of phenomena. For example, Nash equilibia are notoriously inaccurate in predicting how people actually behave in many games (e.g. Prisoners' Dilemma, ultimatum game, public goods game, and so on). And adding behavioral assumptions/relaxing the rationality assumption seemed like a natural way to "fix" existing models to increase their predictive power. Hence we have the rising popularity of behavioral economics these days. Still, these "behavioral patches" are often met with skepticism from of many traditional "rationalists". See this post for example.
