The Rational Expectations Hypothesis (REH) is an hypothesis about aggregate expectations. I believe it is illuminating to post here a lengthy quote (part 2) from Muth (1961) paper where REH originated (bold letters are our emphasis):
2. THE "RATIONAL EXPECTATIONS" HYPOTHESIS
Two major conclusions from studies of expectations data are the following:
1. Averages of expectations in an industry are more accurate than naive models and as accurate as elaborate equation systems, although
there are considerable cross-sectional differences of opinion.
2. Reported expectations generally underestimate the extent of changes that actually take place.
In order to explain these phenomena, I
should like to suggest that expectations, since they are informed
predictions of future events, are essentially the same as the
predictions of the relevant economic theory (We show in Section 5 that
the hypothesis is consistent with these two phenomena). At the risk of
confusing this purely descriptive hypothesis with a pronouncement as
to what firms ought to do, we call such expectations "rational." It is
sometimes argued that the assumption of rationality in economics leads
to theories inconsistent with, or inadequate to explain, observed
phenomena, especially changes over time (e.g., Simon 1959). Our
hypothesis is based on exactly the opposite point of view: that
dynamic economic models do not assume enough rationality.
The hypothesis can be rephrased a little more precisely as follows: that
expectations of firms (or, more generally, the subjective probability
distribution of outcomes) tend to be distributed, for the same
information set, about the prediction of the theory (or the
"objective" probability distributions of outcomes).
The hypothesis asserts three things: (1) Information is scarce, and the economic
system generally does not waste it. (2) The way expectations are
formed depends specifically on the structure of the relevant system
describing the economy. (3) A "public prediction," in the sense of
Grunberg and Modigliani (1954), will have no substantial effect on the
operation of the economic system (unless it is based on inside
information). This is not quite the same thing as stating that the
marginal revenue product of economics is zero, because expectations of
a single firm may still be subject to greater error than the theory.
It does not assert that the scratch work of entrepreneurs resembles
the system of equations in any way; nor does it state that predictions
of entrepreneurs are perfect or that their expectations are all the
same. ...
I believe that it should be clear from the above that:
1) REH is not an assertion about each separate individual, but about the properties of the "prevailing" expectation produced by the black-box combination of individual expectations. In other words the REH is assumed, without really making any assumptions about individual rationality.
2) It has as much to do with the "internal consistency" of the economic model itself, because by construction and without any economic assumptions, $E(X\mid I) = X+ e,\; E(e\mid I) =0 $.
The fact that the predominant economic model framework has been that of the "representative" (identical) consumer, nevertheless blurred the distinction between the aggregate expectation, and individual expectations on aggregate variables. This provided shallow "micro-foundations" to the REH, (shallow because it is not really micro-founded, that which essentially assumes away the need to aggregate), but also, it moved the debate into the arena of individual expectations formation and whether individuals use information efficiently or not, which raised valid objections as those mentioned in the answer by @EnergyNumbers.
But really, at the individual level, the hypothesis that individuals use the mathematical expected value comes essentially from Expected Utility theory, that predates the Rational Expectations, and has a debate on its own (also here in Economics.SE)
Another set of "arguments against" the REH (which gave very interesting literature), was collected early on in the book "Individual forecasting and aggregate outcomes - Rational Expectations examined" 1983 R. Frydman and E. Phelps (ed). Of which I mention two:
1) Being an equilibrium concept, REH requires co-ordination of expectations formation (which is really not that realistic) or properties of Nash-equilibrium: this last insight gave us "Eductive Expectations" and some really thoughtful works by Roger Guesnerie.
2) The second one, which became rather more widely spread than Eductive Expectations, is "Adaptive Learning" (see "Learning and Expectations in Macroeconomics" By Evans and Honkapohja, 2001).
Adaptive Learning pointed out that REH assumes that economic agents know the structure of their environment perfectly. So in Adaptive Learning models we have the first systematic approach to model uncertainty : as economists, so economic agents do not know the environment perfectly, and they have to estimate it and learn it gradually (hence "adaptive learning"). In this strand of literature, "learning" is done through econometric methods, mainly least-squares (which is a very intuitive least-distance mathematical approximation method). Roughly speaking, here agents' expectations are not the expected values, but the estimated expected values. This creates much more interesting and realistic dynamics, that some times may converge (someday) to an REH equilibrium (which makes Adaptive Learning a "selection mechanism" for the sometimes multiple REH equilibira), or to some other point, not predicted by REH.
Research into the issue of aggregate expectations formation and modeling is currently exploding, see for example another Frydman & Phelps (ed.) book, "Rethinking Expectations" (2012), in parallel with the emerging "Post-Walrasian" direction in Macroeconomics (see D. Colander (ed). Post-Walrasian Macroeconomics 2006).