More than one Bellman Equation

I'm attending to my first dynamic optimization course, and what I don't fully graps yet is that sometimes we have to use more than one bellman equation.

How do you realize that? I mean how do you know when your problem solution require more than one Bellman equation?

For example this problem taken from Sargent's Recursive Macroeconomic Theory 2nd edition.

An unemployed worker receives each period a wage offer w drawn from the distribution F(w). The worker has to choose whether to accept the job – and therefore to work forever – or to search for another offer and collect c in unemployment compensation. The worker who decides to accept the job must choose the number of hours to work in each period. The worker chooses a strategy to maximize

$$E\Sigma_{t=0}^{\infty}\beta^{t}u(y_t,l_t)$$

and $$y_t=c$$ if the worker is unemployed, and $$y_t=w(1-l_t)$$ if the worker is employed and works $$(1-l_t)$$ with $$l_t$$ leisure and $$0

Analyze the worker’s problem. Argue that the optimal strategy has the reservation wage property. Show that the number of hours worked is the same in every period.

The solution's manual goes like this for the part of stating the Bellman equations:

Let s be the vector of state variables. We choose $$s=(w,0)$$ where $$w$$ is the wage offer and $$0=E$$ if the worker is employed and $$0=U$$ if the worker is unemployed. Consider first the situation of an employed worker. Bellman’s equation is:

$$v(w,E)= max \{u[w(1-l),l]+\beta v(w,E)\}$$

and for unemployed worker:

$$v(w,U)= max \{v(w,E);u[c,1]+\beta\int v(w',E)DF(w')\}$$

So being more concrete. Why the solutions requires two bellman equations and how do you realize that when reading the problem?

For example my first guess when trying to solve without looking the solutions I wrote:

$$v(w,E)= max \{u[w(1-l),l];u[c,1]+\beta\int v(w',E)DF(w')\}$$

Why is this different?

• "Let s be the state variable" is somewhat misleading. Strictly speaking, the state that enter agent's decision here is a pair (wage offered, employment status). Value functions are, in general, functions of the state. For example, if the state space is a subset of $\mathbb{R}^2$, then the value function is a function, say, V(x,y). Here the second variable is binary (E or U), which gives the appearance of "2 equations". Jun 16, 2020 at 20:20
• Thank you @Michael, so actually there isn't two equations? How do you realize that the emplyoment status is a state variable? Jun 16, 2020 at 20:22
• Not disputing that here there are two equations and two functions. In the case that one of the state coordinates is discrete and others non-discrete, you would multiple equations indexed by the discrete variable, so to speak. Jun 16, 2020 at 20:37
• Here viewing the solution as two functions, one each for E and U, can be informative. If you look at the second equation v(w, U) = ..., it tells you that the w* where the two functions intersect is where agent is indifferent between E and U. If w < w*, agent chooses U. If w > w*, agent choose E. There also tells you E/U should be part of the state. Jun 16, 2020 at 20:44
• The state (vector, if you'd like) consists of all the variable that determines the scenario faced by the agent (i.e. his optimization problem). Here, given the same w, your optimization problem is different if you were employed than if you were unemployed, therefore your employment status is part of the state. Jun 16, 2020 at 21:07

(The second equation for the value function of the unemployed should be $$v(w,U)= \max \{v(w,E); \,u[c,1]+\beta\int v(w', U) dF(w')\}. \quad (*)$$ )

...how do you know when your problem solution require more than one Bellman equation?

Whenever the state space of the problem contains discrete coordinates, there would be "multiple" value functions, indexed by the discrete coordinates. Here the state space is $$[0, \infty) \times \{E, U\}$$. The second coordinate is discrete. So there are "two" value functions $$v(w, E)$$ and $$v(w, U)$$.

How do you realize that?

Any variable that determines the optimization problem faced by the agent is part of the state. In this particular example, an unemployed agent has the option to switch to being employed. An employed agent has no choice but to stay in the job---"...to work forever". The choice set, therefore the decision problem, faced by the agent are different depending on his employment status. This tells you that employment status is part of the state.

(Note that, if the employed worker has the option to leave the job, the decision problem faced by the agent are still different depending on his employment status. In that case you would still have two value functions and they would be "intertwined", so to speak.)

Argue...the optimal strategy has the reservation wage property.

This is more or less immediate from the equation $$(*)$$. The reservation wage $$w^*$$ is given by $$v(w^*,E) = u[c,1]+\beta\int v(w', U) dF(w').$$ At $$w = w^*$$, agent is indifferent, $$v(w^*, U) = v(w^*, E)$$. You would expect that for $$w < w^*$$. $$v(w,U) = u[c,1]+\beta\int v(w', U) dF(w') > v(w,E).$$

This is a typical option-exercise problem. The difference $$v(w, U) - v(w,E)$$ when $$w < w^*$$ is the option value. If the offered wage $$w$$ is too low, the agent would rather keep the option and not exercise it.

(This is an American-type option, which can be exercised anytime. Same "exercise threshold" phenomenon occurs in the continuous-time setting, which is sometimes more convenient. There the threshold is given by the smooth-pasting condition.)

The general formulation of the Bellman equation is $$V(s) = \max_{c \in \mathcal{C}(s)} \int_{\mathcal{S}} V(s') dF(s'; s, c),$$ where $$\mathcal{C}(s)$$ is the choice set faced by the agent at state $$s$$ and $$s' \mapsto dF(s'; s, c)$$ is the Markov transition kernel if agent makes choice $$c$$ at state $$s$$. (For notational simplicity, assume no period utility/discounting/etc. The discussion would not change.) Therefore, by definition, any variable that determines the optimization problem faced by the agent is part of the state.
• Right now the Bellman equation for v(w, E) does not contain v(w, U) on the RHS, because there's no option to switch associated with E. If an E worked has the option to switch to U, you would get for v(w,E) an equation that is similar to $(*)$. So v(w,E) and v(w,U) would both appear on the RHS of each other's equations. Such pair of equations are actually quite common in the real options setting. Jun 18, 2020 at 20:51