More than one Bellman Equation

Question

I'm attending to my first dynamic optimization course, and what I don't fully graps yet is that sometimes we have to use more than one bellman equation.

How do you realize that? I mean how do you know when your problem solution require more than one Bellman equation?

For example this problem taken from Sargent's Recursive Macroeconomic Theory 2nd edition.

An unemployed worker receives each period a wage offer w drawn from the distribution F(w). The worker has to choose whether to accept the job – and therefore to work forever – or to search for another offer and collect c in unemployment compensation. The worker who decides to accept the job must choose the number of hours to work in each period. The worker chooses a strategy to maximize

$E\Sigma_{t=0}^{\infty}\beta^{t}u(y_t,l_t)$

and $y_t=c$ if the worker is unemployed, and $y_t=w(1-l_t)$ if the worker is employed and works $(1-l_t)$ with $l_t$ leisure and $0<l_t<1$

Analyze the worker’s problem. Argue that the optimal strategy has the reservation wage property. Show that the number of hours worked is the same in every period.

The solution's manual goes like this for the part of stating the Bellman equations:

Let s be the vector of state variables. We choose $s=(w,0)$ where $w$ is the wage offer and $0=E$ if the worker is employed and $0=U$ if the worker is unemployed. Consider first the situation of an employed worker. Bellman’s equation is:

$v(w,E)= max \{u[w(1-l),l]+\beta v(w,E)\}$

and for unemployed worker:

$v(w,U)= max \{v(w,E);u[c,1]+\beta\int v(w',E)DF(w')\}$

So being more concrete. Why the solutions requires two bellman equations and how do you realize that when reading the problem?

For example my first guess when trying to solve without looking the solutions I wrote:

$v(w,E)= max \{u[w(1-l),l];u[c,1]+\beta\int v(w',E)DF(w')\}$

Why is this different?

Thanks in advance.

Answer 1

(The second equation for the value function of the unemployed should be $$ v(w,U)= \max \{v(w,E); \,u[c,1]+\beta\int v(w', U) dF(w')\}. \quad (*) $$ )

...how do you know when your problem solution require more than one Bellman equation?

Whenever the state space of the problem contains discrete coordinates, there would be "multiple" value functions, indexed by the discrete coordinates. Here the state space is $[0, \infty) \times \{E, U\}$. The second coordinate is discrete. So there are "two" value functions $v(w, E)$ and $v(w, U)$.

How do you realize that?

Any variable that determines the optimization problem faced by the agent is part of the state. In this particular example, an unemployed agent has the option to switch to being employed. An employed agent has no choice but to stay in the job---"...to work forever". The choice set, therefore the decision problem, faced by the agent are different depending on his employment status. This tells you that employment status is part of the state.

(Note that, if the employed worker has the option to leave the job, the decision problem faced by the agent are still different depending on his employment status. In that case you would still have two value functions and they would be "intertwined", so to speak.)

Argue...the optimal strategy has the reservation wage property.

This is more or less immediate from the equation $(*)$. The reservation wage $w^*$ is given by $$ v(w^*,E) = u[c,1]+\beta\int v(w', U) dF(w'). $$ At $w = w^*$, agent is indifferent, $v(w^*, U) = v(w^*, E)$. You would expect that for $w < w^*$. $$ v(w,U) = u[c,1]+\beta\int v(w', U) dF(w') > v(w,E). $$

This is a typical option-exercise problem. The difference $v(w, U) - v(w,E)$ when $w < w^*$ is the option value. If the offered wage $w$ is too low, the agent would rather keep the option and not exercise it.

(This is an American-type option, which can be exercised anytime. Same "exercise threshold" phenomenon occurs in the continuous-time setting, which is sometimes more convenient. There the threshold is given by the smooth-pasting condition.)

General Comments

The general formulation of the Bellman equation is $$ V(s) = \max_{c \in \mathcal{C}(s)} \int_{\mathcal{S}} V(s') dF(s'; s, c), $$ where $\mathcal{C}(s)$ is the choice set faced by the agent at state $s$ and $s' \mapsto dF(s'; s, c)$ is the Markov transition kernel if agent makes choice $c$ at state $s$. (For notational simplicity, assume no period utility/discounting/etc. The discussion would not change.) Therefore, by definition, any variable that determines the optimization problem faced by the agent is part of the state.