# Relation between Markov Perfect Nash Equilibrium and Markovian evolution of the state

I am studying dynamic games and I'm fundamentally confused about the relation between Markov Perfect Nash Equilibrium and Markovian evolution of the state. Before illustrating my doubt, let me describe the basic framework.

Consider a game played by N players that we index by $$i=1,...,N$$. Time is discrete and indexed by $$t$$. In each period $$t$$, every player $$i$$ chooses an action $$a_{it}\in \mathcal{A}$$, where $$\mathcal{A}$$ is finite and fixed across time/players for simplicity. $$a_t\equiv (a_{1t},..., a_{Nt})$$ is the vector of all players' actions in period t. In each period $$t$$, each player $$i$$ gets a payoff, $$\pi_i(a_t, x_t)$$, that depends on $$a_t$$ and on a vector of common knowledge state variables, $$x_t$$, with support $$\mathcal{X}$$. Each player $$i$$ chooses the actions that maximise his expected and discounted flow of profits $$E_t(\sum_{s=0}^\infty \beta_i \pi(a_t, x_t))$$ where $$\beta_i$$ is the discount factor.

It is common to make two assumptions in applied work:

1. The players play Markov Perfect Nash Equilibrium (MPNE). That is, their strategies in period $$t$$ are functions only of the payoff-relevant state variables at the same period.

2. The vector of state variables, $$x_t$$, follows a first order controlled Markov process with transition CDF $$F(x_{t+1}| x_t, a_t)$$.

My understanding is that the payoff-relevant state variables (mentioned in ass. 1) are the state variables explicitly entering $$\pi$$. In this case, just $$x_t$$. Therefore, the players' strategies are $$\alpha\equiv \{\alpha_i(x_t): i=1,...,N \text{ and } x_t\in \mathcal{X}\}$$.

$$\alpha$$ is a MPNE if it satisfies $$(*) \quad \alpha_i(x_t)= \text{argmax}_{a_{it}\in \mathcal{A}} \Big\{ \pi_i (a_{it}, \{\alpha_j(x_t)\}_{j\neq i}, x_t)+\beta_i \int V_i^\alpha(x_{t+1}) d F(x_{t+1}| x_t, a_{it}, \{\alpha_j(x_t)\}_{j\neq i}) \Big\}$$ for each player $$i$$ and state $$x_t$$, where $$V_i^\alpha$$ is the value function uniquely solving the Bellman equation: $$(**) \quad V_i^\alpha(x_t)=\max_{a_{it}\in \mathcal{A}} \Big\{ \pi_i (a_{it}, \{\alpha_j(x_t)\}_{j\neq i}, x_t)+\beta_i \int V_i^\alpha(x_{t+1}) d F(x_{t+1}| x_t, a_{it}, \{\alpha_j(x_t)\}_{j\neq i}) \Big\}$$

Questions:

• I am confused about the relation between ass. 1 and 2. In particular, it seems to me that ass. 2 is "necessary" for ass. 1. In fact, suppose that $$x_t$$ follows a second order controlled Markov process with transition CDF $$F(x_{t+1}| x_t, a_t, x_{t-1}, a_{t-1})$$. Then, $$x_{t-1}, a_{t-1}$$ would appear in $$(*)$$ and $$(**)$$, in addition to $$x_t$$. In turn, this would invalidate the fact that the players' strategies can depend on $$x_t$$ only.

• Here, perhaps, I'm misunderstanding the definition of "payoff-relevant state variables". Are these the state variables explicitly entering $$\pi$$ (as I initially thought), or are these the state variables that matter for the evolution of the state? The latter interpretation, if correct, would clear any confusion: suppose, for instance, that $$x_t$$ follows a second order controlled Markov process with transition CDF $$F(x_{t+1}| x_t, a_t, x_{t-1}, a_{t-1})$$; in this case, the payoff-relevant state variables would be $$x_t, x_{t-1}, a_{t-1}$$. ; in turn, a MPNE would be a strategy $$\alpha_i(x_t, x_{t-1}, a_{t-1})$$ for each player $$i$$.

• Suppose that 2 holds. Could you show me how does the definition of MPNE differ from a classic definition of subgame perfect Nash equilibrium? In particular, if the state evolves as a first order Markov, why should a player condition his strategies on past history?

Q1. Assumptions 1 and 2 are independent of one another. Assumption 2 is an assumption of the fundamentals of the game, the setup if you will, and have no restrictions of the solutions. Its merely a description of the game.

Assumption 1 on the other hand, is an assumption on the choice of the class of equilibria we are interested in. I agree that Assumption 2 makes it natural to look at Markov perfect equilibria, but there is no such inherent need to do so.

EDIT: See Micheal's comment below for a better answer.

Q2. Your latter interpretation is the one I would go for.

Q3. A SPNE here would be a time and history dependent strategy. So $$a_{it}: \times_{i = 1}^{t} (\mathcal{A} \times \mathcal{X})_i \rightarrow \mathcal{A}$$ i.e. the stategy at period $$t$$ under state $$x_t$$ depends on the entire history of realised states and actions played.

Its clear a Markov strategy is a special case of the above.

• Assumptions 1 and 2 are not quite independent. If the state variable follows a first-order Markov process and every other player plays according to a Markovian strategy, then the decision problem a player faces at each state will be Markovian too and allow for Markovian best responses. Without the state following a first-order Markov process, there is little hope that SPNE in Markovian strategies exist. Feb 22, 2022 at 22:19
• That is what I thought too. Feb 22, 2022 at 22:32
• I've edited my answer to reflect the better insight provided by Michael. Feb 22, 2022 at 23:32