3
$\begingroup$

I'm attending to my first dynamic optimization course, and what I don't fully graps yet is that sometimes we have to use more than one bellman equation.

How do you realize that? I mean how do you know when your problem solution require more than one Bellman equation?

For example this problem taken from Sargent's Recursive Macroeconomic Theory 2nd edition.

An unemployed worker receives each period a wage offer w drawn from the distribution F(w). The worker has to choose whether to accept the job – and therefore to work forever – or to search for another offer and collect c in unemployment compensation. The worker who decides to accept the job must choose the number of hours to work in each period. The worker chooses a strategy to maximize

$E\Sigma_{t=0}^{\infty}\beta^{t}u(y_t,l_t)$

and $y_t=c$ if the worker is unemployed, and $y_t=w(1-l_t)$ if the worker is employed and works $(1-l_t)$ with $l_t$ leisure and $0<l_t<1$

Analyze the worker’s problem. Argue that the optimal strategy has the reservation wage property. Show that the number of hours worked is the same in every period.

The solution's manual goes like this for the part of stating the Bellman equations:

Let s be the vector of state variables. We choose $s=(w,0)$ where $w$ is the wage offer and $0=E$ if the worker is employed and $0=U$ if the worker is unemployed. Consider first the situation of an employed worker. Bellman’s equation is:

$v(w,E)= max \{u[w(1-l),l]+\beta v(w,E)\}$

and for unemployed worker:

$v(w,U)= max \{v(w,E);u[c,1]+\beta\int v(w',E)DF(w')\}$

So being more concrete. Why the solutions requires two bellman equations and how do you realize that when reading the problem?

For example my first guess when trying to solve without looking the solutions I wrote:

$v(w,E)= max \{u[w(1-l),l];u[c,1]+\beta\int v(w',E)DF(w')\}$

Why is this different?

Thanks in advance.

$\endgroup$
  • 2
    $\begingroup$ "Let s be the state variable" is somewhat misleading. Strictly speaking, the state that enter agent's decision here is a pair (wage offered, employment status). Value functions are, in general, functions of the state. For example, if the state space is a subset of $\mathbb{R}^2$, then the value function is a function, say, V(x,y). Here the second variable is binary (E or U), which gives the appearance of "2 equations". $\endgroup$ – Michael Jun 16 at 20:20
  • $\begingroup$ Thank you @Michael, so actually there isn't two equations? How do you realize that the emplyoment status is a state variable? $\endgroup$ – Martin Mendina Jun 16 at 20:22
  • $\begingroup$ Not disputing that here there are two equations and two functions. In the case that one of the state coordinates is discrete and others non-discrete, you would multiple equations indexed by the discrete variable, so to speak. $\endgroup$ – Michael Jun 16 at 20:37
  • $\begingroup$ Here viewing the solution as two functions, one each for E and U, can be informative. If you look at the second equation v(w, U) = ..., it tells you that the w* where the two functions intersect is where agent is indifferent between E and U. If w < w*, agent chooses U. If w > w*, agent choose E. There also tells you E/U should be part of the state. $\endgroup$ – Michael Jun 16 at 20:44
  • 1
    $\begingroup$ The state (vector, if you'd like) consists of all the variable that determines the scenario faced by the agent (i.e. his optimization problem). Here, given the same w, your optimization problem is different if you were employed than if you were unemployed, therefore your employment status is part of the state. $\endgroup$ – Michael Jun 16 at 21:07
1
$\begingroup$

(The second equation for the value function of the unemployed should be $$ v(w,U)= \max \{v(w,E); \,u[c,1]+\beta\int v(w', U) dF(w')\}. \quad (*) $$ )

...how do you know when your problem solution require more than one Bellman equation?

Whenever the state space of the problem contains discrete coordinates, there would be "multiple" value functions, indexed by the discrete coordinates. Here the state space is $[0, \infty) \times \{E, U\}$. The second coordinate is discrete. So there are "two" value functions $v(w, E)$ and $v(w, U)$.

How do you realize that?

Any variable that determines the optimization problem faced by the agent is part of the state. In this particular example, an unemployed agent has the option to switch to being employed. An employed agent has no choice but to stay in the job---"...to work forever". The choice set, therefore the decision problem, faced by the agent are different depending on his employment status. This tells you that employment status is part of the state.

(Note that, if the employed worker has the option to leave the job, the decision problem faced by the agent are still different depending on his employment status. In that case you would still have two value functions and they would be "intertwined", so to speak.)

Argue...the optimal strategy has the reservation wage property.

This is more or less immediate from the equation $(*)$. The reservation wage $w^*$ is given by $$ v(w^*,E) = u[c,1]+\beta\int v(w', U) dF(w'). $$ At $w = w^*$, agent is indifferent, $v(w^*, U) = v(w^*, E)$. You would expect that for $w < w^*$. $$ v(w,U) = u[c,1]+\beta\int v(w', U) dF(w') > v(w,E). $$

This is a typical option-exercise problem. The difference $v(w, U) - v(w,E)$ when $w < w^*$ is the option value. If the offered wage $w$ is too low, the agent would rather keep the option and not exercise it.

(This is an American-type option, which can be exercised anytime. Same "exercise threshold" phenomenon occurs in the continuous-time setting, which is sometimes more convenient. There the threshold is given by the smooth-pasting condition.)

General Comments

The general formulation of the Bellman equation is $$ V(s) = \max_{c \in \mathcal{C}(s)} \int_{\mathcal{S}} V(s') dF(s'; s, c), $$ where $\mathcal{C}(s)$ is the choice set faced by the agent at state $s$ and $s' \mapsto dF(s'; s, c)$ is the Markov transition kernel if agent makes choice $c$ at state $s$. (For notational simplicity, assume no period utility/discounting/etc. The discussion would not change.) Therefore, by definition, any variable that determines the optimization problem faced by the agent is part of the state.

| improve this answer | |
$\endgroup$
  • $\begingroup$ "Note that, if the employed worker has the option to leave the job, the decision problem faced by the agent are still different depending on his employment status. In that case you would still have two value functions and they would be "intertwined", so to speak.)" What do you mean by that? I don't know what "intertwined" is. $\endgroup$ – Martin Mendina Jun 18 at 20:14
  • $\begingroup$ Right now the Bellman equation for v(w, E) does not contain v(w, U) on the RHS, because there's no option to switch associated with E. If an E worked has the option to switch to U, you would get for v(w,E) an equation that is similar to $(*)$. So v(w,E) and v(w,U) would both appear on the RHS of each other's equations. Such pair of equations are actually quite common in the real options setting. $\endgroup$ – Michael Jun 18 at 20:51
  • $\begingroup$ Thank you I understood that. Going one step further, in that case (Employee can move from E to U) would the BE be equal to (*) for both "workers" (E and U)? $\endgroup$ – Martin Mendina Jun 18 at 23:26
  • $\begingroup$ No, there would be two equations, just as is currently the case---the discussion regarding what the state space is does not change. The difference would be that v(w,E) would reflect the option value of switching to U, whereas it currently does not (because there is no such option). Whenever you see max {value functions for different states} on the RHS of Bellman eqn---here max { v(w,E) ; continuation value in U which involves v(w,U) }, the eqn reflects the optionality of the problem. $\endgroup$ – Michael Jun 18 at 23:39
  • 1
    $\begingroup$ That would depend on the specific model. What is the state space? Ask yourself the question "does the specification of agent's problem---the choice set, the objective function, the constraint---depend on the variable I am looking at?" Is the choice set, the objective function, the constraint different for employed and unemployed? If answer is yes, then it's part of the state. The value function V is a function on the state space, whatever that is. If the state space has a discrete coordinate like E/U, then the Bellman eqn(s) is specified accordingly. For binary state like E/U, "two" eqns. $\endgroup$ – Michael Jun 18 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.