# What is the exact relationship between the efficient-market and random-walk hypotheses?

The efficient-market and random-walk hypotheses are often discussed together as formalizations of the idea that "you can't consistently beat the stock market". But in my experience, they're often used interchangeably, and I've never seen anyone carefully explain the difference. (For example, Wikipedia says that A Random Walk Down Wall Street "argues that asset prices typically exhibit signs of random walk and that one cannot consistently outperform market averages. The book is frequently cited by those in favor of the efficient-market hypothesis." The Wikipedia article on the EMH says "there is a very close link between EMH and the random walk hypothesis" and the one on the RWH says that it "is consistent with the efficient-market hypothesis", but neither spells out a precise relationship.)

What is the exact distinction? Is either hypothesis a special case of the other? Put another way, could one imagine a stock market in which either hypothesis holds but the other does not? The EMH page says "Samuelson published a proof showing that if the market is efficient prices will show random-walk behavior", which seems to imply that the EMH is a special case of the RWH - so that one could imagine a stock market in which RWH holds but not EMH, but not the opposite. Is this correct?

I just want to build on @Alecos's nice answer. I'll restate some of what he said and add a few other details.

## tl;dr

• They don't have anything to do with each other. The assumption of efficient markets does not imply that asset prices follow a random walk and the random-walk assumption does not imply efficient markets.
• Due the "joint hypothesis theorem," the efficient markets hypothesis (EMH) doesn't actually have any testable content. This means that the EMH not only doesn't imply that prices are a random walk, but it means that the EMH doesn't imply much of anything at all!

In the following,

1. I'll formalize what the EMH says,
2. use this formulation to describe the "joint hypothesis theorem,"
3. talk about the difference between prices "containing all information" and prices "reflecting all information,"
4. and talk about how conditional probabilities relate to random walks.
5. I'll also discuss how the concept of a random walk does show up when we make additional assumptions.
6. However, as we'll see, this still does not imply that, say, stock returns are unpredictable.

## 1. What does the Efficient Markets Hypothesis formally say?

In this section, I draw from Fama's papers directly. Here, I pull from his book "Foundations of Finance." To get the full story, check out Fama's website. You'll find some links to some books of his. The following comes from the first two sections of chapter 5 of his book, "Foundations of Finance."

The EMH is about information. This can be formally represented by information sets that, in probability theory, are represented by "sigma-algebras." Suppose that all information at time $t-1$ is contained in the information set $I_{t-1}$ and let the information set possessed by the market be $I^m_{t-1}$. By definition, all we can say is that the information set possessed by the market is less than (a subset of) all information available at that time. That is, $I^m_{t-1} \subset I_{t-1}$. The EMH posits, however, that these two information sets are in fact equal---that $I^m_{t-1} = I_{t-1}$.

With this definition in mind, it is still not quite clear what this implies. This is made more clear in the "joint hypothesis theorem" (or, joint hypothesis problem).

## 2. What is the Joint Hypothesis Theorem?

The joint hypothesis theorem (JHT), loosely stated, is a theorem that states that you cannot test market efficiency without describing how the market uses its information (e.g., to form prices). That is, you can test market efficiency by simply assuming that the market behaves a certain way. Or, you can assume that markets are efficient and infer how markets behave. But, you cannot do both. Testing one requires assuming the other.

I have discussed this theorem to in another post on this site. I will reproduce some (a lot?) of it here.

Tests of the efficient markets hypothesis must always include a model of how the market forms prices. One of Fama's big contributions was that you cannot separate these two things in a test. Tests of efficient markets and models of price formations are inherently linked. So, in this case, the assumption of constant real interest rates is merely an assumption of how the market forms prices. Only after assuming this, can we test market efficiency. You might disagree with the model---more sophisticated models will repeat this experiment in that way---but the point is that you have to assume some model for price formation.

The argument that he makes in that chapter is approximately the following (borrowing his notation, but changing $\phi$ to $I$). The point that he makes is that we want to test if the information sets are equal $I_{t-1}^m = I_{t-1}$, where $I_{t-1}^m$ is the information that the market possesses. But because we can't test this directly, we would like to test whether the distributions of prices are the same $$f_m(p_1,...,p_n \mid I_{t-1}^m) = f(p_1,..., p_n \mid I_{t-1}).$$ However, this is impossible too. The equality possesses no testable content because we only observe $f(p_1,..., p_n \mid I_{t-1})$ and not $f_m(p_1,...,p_n \mid I_{t-1}^m)$ (see the top of page 137 of the linked chapter). I do not observe the latter because I do not know what $I_{t-1}^m$, except that $I_{t-1}^m \subseteq I_{t-1}$, and I do not know how the market uses this information. For this reason, we specify a model for how the market takes information and turns it into prices. Thus, we specify $f_m$ ourselves (in turn, also specifying what information $I_{t-1}^m$ the market uses). That is, we specify what data the market uses and the way in which it uses that data.

On page 134, Fama says

the statement that prices in an efficient market "fully reflect" available information conveys the general idea of what is meant by market efficiency, but the statement is too general to be testable. Since the goal is to test the extent to which the market is efficient, the proposition must be restated in a testable form. ... this requires a more detailed specification of the process of price formation, one that gives testable content to the term "fully reflect."

The key take-away from this is that the EMH is about information. But it doesn't tell use about the way the market uses that information to determine prices (of $n$ assets $p_1, ..., p_n$). The behavior is given by $f_m$, which we have not specified. The assumption of a random walk in prices is, in part, as assumption about $f_m$. It is not so much an assumption about $I^m_{t-1}$. To get to something resembling the random walk theory, we need to add assumptions about the market being competitive.

## 3. Is there a difference between a price "reflecting all information" and "containing all information"?

Sure, this is a somewhat arbitrary semantic difference. Nonetheless, I think that there is an important point to be made here. So let's try to be a little more formal with it. Let's take a look a version the random walk hypothesis. Consider the model $$p_t = d + p_{t-1} + \epsilon_{t},$$ where $d$ is a fixed drift rate and $\epsilon_t$ is a sequence of mean zero iid shocks. I believe that it is a common mistake to claim that the EMH implies that $p_{t-1}$ "contains all available information." Why? Because this claim seems to imply that the information set $I_{t-1}$ can be spanned by (or generated by) the single variable $p_{t-1}$. However, as we discussed in the previous sections, the EMH does not make this claim. Instead, the EMH makes the claim that the price $p_{t-1}$ "reflects all information" in the sense that the behavior of prices is determined by $f_m$ conditioned on $I_{t-1}$. That is, the distribution of $n$ assets over the next period is $f_m(p_1,..., p_n | I^m_{t-1})$.

With this in mind, it seems clear and likely in many situations that for most random variables $X_t$, including future prices, $$E[X_t | p_{t-1}] \neq E[X_t | I_{t-1}^m].$$ However, this ultimately depends, again, on how the market uses the information to form prices, as encoded in $f_m$. It very well may be that under some assumptions about $f_m$ that $$E[p_t | p_{t-1}] = E[p_t | I_{t-1}^m].$$

## 4. What about the point that expectations conditional on all available information is the best predictor (in the sense on minimizing mean squared errors)?

Suppose the market were to compute the conditional expectation of tomorrows prices based on all available information. Then we could write that $p_t = E[p_t | I^m_{t-1}] + u_t$, where $E[u_t | I^m_{t-1}] = 0$. If we assume the EMH, then $I_{t-1} = I^m_{t-1}$ and thus $E[u_t | I_{t-1}] = 0$ as well.

This is true. However, this shouldn't be taken to mean that (1) any one person has all the information of the market necessary to make this calculation or (2) that this implies anything about the behavior of prices in the market. This is simply a calculation of a conditional expected value.

On the first point (1), recall that the EMH, like the assumption of Rational Expectations , is an assumption about aggregates. From Wikipedia, it assumes "that on average the population is correct (even if no one person is). The markets are a machine that aggregates individual's incomplete information and produces something that reflects all information. See also this answer about Rational Expectations.

On the second point (2), the calculation of this conditional expectation is just that. A calculation. It doesn't say anything about how the market behaves or what it will do with this calculation. This behavior is encoded in $f_m$.

## 5. The random walk hypothesis does show up under the assumption of competitive markets (e.g., no arbitrage).

If we assume that there is no arbitrage (type 1 and type 2), this implies the existence of a strictly positive stochastic discount factor $\{\Lambda_t\}$ (state prices). Together with the EMH, we get $$\Lambda_t p_t = E\left[ \sum_{\tau=t+1}^\infty \Lambda_\tau d_\tau | I_t \right].$$ We can rewrite this as $$\Lambda_t p_t = E[\Lambda_{t+1}(p_{t+1} + d_{t+1})| I_t].$$ That is, discounted price changes, after correction for dividends are unpredictable. Note, however, that this leaves the door open for the possibility that prices or returns themselves can be predictable. And, as a matter of fact, returns do seem to be predictable to substantial degree at business cycle frequencies. See the paper "Discount Rates," by John Cochrane. (Click here for the YouTube presentation cued to the right spot.) However, under these assumptions, discounted price changes, after correction for dividends cannot be predicted.

## 6. Under the Efficient Markets Hypothesis and the assumption of no arbitrage, can changes in the stock price be predictable?

As we discussed, discounted price changes, after correction for dividends are unpredictable under these assumptions. However, it is still possible for returns to be predictable. I'm going to leave most of this discussion to the following three YouTube videos by John Cochrane (from his former Coursera course):

To keep things short, the idea behind predictability can be seen by an approximation of a special case (here we use a simple version of the Consumption Capital Asset Pricing model). This is from p.23 of Cochrane's book Asset Pricing (Revised Edition). In this picture, $R$ are returns $R^f$ is the risk-free rate, $m_t$ is the stochastic discount factor, and $\gamma_t$ is risk aversion at time $t$. The conditional expected returns of future returns varies. It could vary based on any of the right-hand side variables. Empirical evidence seems to indicate that the variation is most plausibly coming from changes in risk aversion $\gamma_t$. If people become more risk averse, which they appear to do during recessions (seemingly for good reason), then they will demand higher expected returns per unit of risk that they take on. This means that information about the variables that determine risk aversion today give us information about tomorrow's returns.

Note that this doesn't imply a violation of no arbitrage. Sure, we can predict higher returns. But this simply reflects the fact that the "price of risk" has gone up. The amount of returns that I demand per unit of risk is changing. My prediction of higher returns is not gaining me any free lunch. It's just reflecting changes in my preferences.

## Extra

For a nice discussion of the efficient market's hypothesis, the joint hypothesis problem, and how it fits into the modern stochastic discount framework, check out the book "The Fama Portfolio." I drew partly from the essay "Efficient Markets and Empirical Finance." In this essay, among other things, they talk about how, in the stochastic discount framework (SDF), $$\Lambda_t p_t = E[\Lambda_{t+1}(p_{t+1} + d_{t+1})| I_t] = \sum_s \pi(s) \Lambda_{t+1}(s)(p_{t_1}(s) + d_{t+1}(s)),$$ the efficient markets hypothesis is about the probabilities $\pi(s)$ and the market behavior is encapsulated in SDF $\Lambda_{t+1}(s)$. This makes the joint hypothesis problem clear---we cannot test this theory without assuming something about one or the other. They don't have anything to do with each other.

Use the superscript $e$ to denote "anticipated value" in whatever way it is formed. Then the Efficient Market Hypothesis states

$$X^e_{t+1,market} = E(X_{t+1}\mid I_t)$$

In words, the value anticipated "by the market" equals the true expected value conditional on all information available at time $t$. So first and foremost, the EMH is about what is the predictor that reflects the anticipation of "the market", and it asserts that it is the conditional expected value based on all information available (to state the obvious, there are many other predictors of future values. We can use the median, the mode, etc etc etc).

By the properties of the conditional expectation we have that

$$X_{t+1} = E(X_{t+1}\mid I_t) + u_{t+1},\;\;\; E(u_{t+1}\mid I_t)=0$$

Therefore the EMH claims that the value anticipated by the market, is off-target only by something ($u_{t+1}$) that cannot be predicted as anything other than zero, given the information available. But that is what it means to "use the information efficiently": to extract from it everything that it can possibly tell you.

The "cannot beat the market" assertion is reflected in the part of the EMH that "the market" forms an expectation as though it was a single mind having available, again, all information. This implies that while "inside", "private" information may matter for what individual agents do and accomplish, if we somehow "average/combine" over those individuals each having a different information set, we will obtain the conditional expected value we would have obtained if we have collected all information in one Information set.

Nowhere in the above do we mention, or need to mention, what is the Data Generating Mechanism for $X$. It can be anything.

As you say the RWH is an implication of the EMH, but the other way doesn't necessarily hold. Consider the problem of predicting the value, $X$, of the stock market at time $t+1$. The random walk hypothesis says $$X_{t+1} = X_{t} + \epsilon_t$$ where $X_t$ is the value today and $\epsilon_t$ is a random variable. The EMH says the value tomorrow is the value today plus some random variable that is uncorrelated across $t$.

The EMH is about the $X_t$. It says that all available information is already captured in $X_t$.

But you could imagine a separate example where all people overvalue the stock market by 10 percent for some reason. Let's assume you know with certainty this is going to continue into the future. Then the EMH doesn't hold, stocks are overvalued by 10 percent, but the RWH does. That is, the value of the stock market follows a random walk, but the EMH doesn't hold.

• According to tcd.ie/Economics/assets/pdf/SER/2007/Samuel_Dupernex.pdf and people.duke.edu/~rnau/411rand.htm, $\epsilon_t$ doesn't have to have mean $0$ if $X$ is a "random walk with a drift". I believe that this drift corresponds to the "known information" about the "fundamentals", and it usually changes on timescales much slower than the day-to-day stock price fluctuations that active traders usually try to exploit. The key assumption about $\epsilon_t$ is not that it has mean $0$, but rather that it is uncorrelated across different $t$. Mar 8 '18 at 22:42
• You are absolutely correct. My attempt to avoid talking about Martingales led me to say something completely wrong. For example, the stock market increases in value over time. Indicating epsilon isn’t mean 0. Mar 9 '18 at 4:10