I am studying dynamic games and I'm fundamentally confused about the relation between Markov Perfect Nash Equilibrium and Markovian evolution of the state. Before illustrating my doubt, let me describe the basic framework.
Consider a game played by N players that we index by $i=1,...,N$. Time is discrete and indexed by $t$. In each period $t$, every player $i$ chooses an action $a_{it}\in \mathcal{A}$, where $\mathcal{A}$ is finite and fixed across time/players for simplicity. $a_t\equiv (a_{1t},..., a_{Nt})$ is the vector of all players' actions in period t. In each period $t$, each player $i$ gets a payoff, $\pi_i(a_t, x_t)$, that depends on $a_t$ and on a vector of common knowledge state variables, $x_t$, with support $\mathcal{X}$. Each player $i$ chooses the actions that maximise his expected and discounted flow of profits $$ E_t(\sum_{s=0}^\infty \beta_i \pi(a_t, x_t)) $$ where $\beta_i$ is the discount factor.
It is common to make two assumptions in applied work:
1. The players play Markov Perfect Nash Equilibrium (MPNE). That is, their strategies in period $t$ are functions only of the payoff-relevant state variables at the same period.
2. The vector of state variables, $x_t$, follows a first order controlled Markov process with transition CDF $F(x_{t+1}| x_t, a_t)$.
My understanding is that the payoff-relevant state variables (mentioned in ass. 1) are the state variables explicitly entering $\pi$. In this case, just $x_t$. Therefore, the players' strategies are $\alpha\equiv \{\alpha_i(x_t): i=1,...,N \text{ and } x_t\in \mathcal{X}\}$.
$\alpha$ is a MPNE if it satisfies $$ (*) \quad \alpha_i(x_t)= \text{argmax}_{a_{it}\in \mathcal{A}} \Big\{ \pi_i (a_{it}, \{\alpha_j(x_t)\}_{j\neq i}, x_t)+\beta_i \int V_i^\alpha(x_{t+1}) d F(x_{t+1}| x_t, a_{it}, \{\alpha_j(x_t)\}_{j\neq i}) \Big\} $$ for each player $i$ and state $x_t$, where $V_i^\alpha$ is the value function uniquely solving the Bellman equation: $$ (**) \quad V_i^\alpha(x_t)=\max_{a_{it}\in \mathcal{A}} \Big\{ \pi_i (a_{it}, \{\alpha_j(x_t)\}_{j\neq i}, x_t)+\beta_i \int V_i^\alpha(x_{t+1}) d F(x_{t+1}| x_t, a_{it}, \{\alpha_j(x_t)\}_{j\neq i}) \Big\} $$
Questions:
I am confused about the relation between ass. 1 and 2. In particular, it seems to me that ass. 2 is "necessary" for ass. 1. In fact, suppose that $x_t$ follows a second order controlled Markov process with transition CDF $F(x_{t+1}| x_t, a_t, x_{t-1}, a_{t-1})$. Then, $x_{t-1}, a_{t-1}$ would appear in $(*)$ and $(**)$, in addition to $x_t$. In turn, this would invalidate the fact that the players' strategies can depend on $x_t$ only.
Here, perhaps, I'm misunderstanding the definition of "payoff-relevant state variables". Are these the state variables explicitly entering $\pi$ (as I initially thought), or are these the state variables that matter for the evolution of the state? The latter interpretation, if correct, would clear any confusion: suppose, for instance, that $x_t$ follows a second order controlled Markov process with transition CDF $F(x_{t+1}| x_t, a_t, x_{t-1}, a_{t-1})$; in this case, the payoff-relevant state variables would be $x_t, x_{t-1}, a_{t-1}$. ; in turn, a MPNE would be a strategy $\alpha_i(x_t, x_{t-1}, a_{t-1})$ for each player $i$.
Suppose that 2 holds. Could you show me how does the definition of MPNE differ from a classic definition of subgame perfect Nash equilibrium? In particular, if the state evolves as a first order Markov, why should a player condition his strategies on past history?
