0
$\begingroup$

\begin{align} V(W)=\max\limits_{W'\in[0,W]}\qquad& u(W-W')+\beta V(W')\qquad\forall W \end{align}

$\textbf{My Question}$: Why is the unknown in the Bellman equation $V(W)$ itself? Isn't the choice variable tomorrow's state, $W'$?

$\endgroup$
4
$\begingroup$

The original problem was probably of the form $$\max_{\{W_t\}_{t=1}^\infty}\sum_{t=0}^\infty \beta^t u(W_t-W_{t+1}),$$ $$\mbox{s.t. } W_{t+1}\in[0,W_t] \ \forall \ t, ~~ W_0 \mbox{ given}$$ When formulated that way there is no need to solve for any function $V$, but the infinite elements of the sequence ${\{W_t\}}_{t=1}^\infty$ are all choice variables. Bellman's main insight is that (under some conditions) the problem can be simplified. Instead of solving for an infinite number of variables, you could solve for a single "policy function" that tells you what to do each period as a function of the current state variables, and a "value function" that summarizes what is the value (in terms of today) of inducing each possible state in the future. Even if there is no uncertainty, it is clear that this value function is an endogenous object, since it must somehow capture the future stream of utility as a single function of the future state. In your case, that function is $V(W')$. This is why the Bellman equation: $$V(W)=\max_{W'\in[0,W]} u(W-W')+\beta V(W')$$ has two properties:

1) it lacks the subscripts on the $W$'s, this reflects that you will solve for $W'$ as a function of $W$, i.e. the policy function.

2) has a new object, $V$, that was not in the original problem and it is a function of $W$ (or $W'$) and just by eyeballing the two problems you can see that for them to be equivalent $V$ must be the discounted sum of future payoffs assuming future actions are also optimal. So it is not a choice variable, but definitely, an endogenous object that you need to find when solving the problem.

You can see that the problem of optimally choosing infinitely many objects is transformed into one of choosing a finite number (2) of functions, which are infinite-dimensional objects. Brilliant!

$\endgroup$
  • $\begingroup$ Hi Regio, thanks for the clear explanation. I am getting a firmer grasp on the DP. One additional question is that why does the Bellman equation usually emphasize that the equation must hold for all (feasible) values of $W$? Also, I see that $V(W)$ is the current value function and $V(W')$ is the continuation value function, but how do you mean when you say it is "an endogenous object that you need to... the problem?" Just because it appears on both sides of the Bellman equation? I think what I am confused is that under the max operator $W'$ is choice variable but we say $V(\cdot)$ is the sol $\endgroup$ – Frank Swanton Jun 18 at 8:13
  • 1
    $\begingroup$ Not sure I understand your question. But $V(W)$ is the value of currently having a stock of $W$ (that is why $\beta V(W')$ is the discounted value of having a stock of $W'$ next period). and for you to be able to know how much should you consume today, $u(W-W')$ it is necessary to know what payoff each of these consumptions will imply for tomorrow. Note that different consumptions induce different $W'$. That's why $V(\cdot)$ must be valid for all values of $W$. $\endgroup$ – Regio Jun 18 at 17:03
  • 1
    $\begingroup$ Maybe it will be easier for you to see the max operator as some function $F$. then the Bellman equation is of the form $V(W)=F(\cdot, V(W'))$, where $V$ is the unique function that satisfies this equality. (the uniqueness requires some conditions, but in most economics problems you will have uniqueness. Because $F$ is a max operator it can be challenging to find $V$, but there are some standard techniques that can be helpful. $\endgroup$ – Regio Jun 18 at 17:07
  • $\begingroup$ Thanks for the comments Regio. It helps a lot. Do you have any reference material where I can see an example of a problem that has a full-fledged version of solving a DP program including the lagrangean, envelope conditions, and etc? I want to work out a full version if you have a credible reference? thanks $\endgroup$ – Frank Swanton Jun 20 at 2:09
  • $\begingroup$ A classic reference is Thomas Sargent's book. However if you do a quick search with "Dynamic programming macroeconomics" you can find many lecture notes of introductory macro courses at the graduate level that will likely be helpful. $\endgroup$ – Regio Jun 20 at 20:47
1
$\begingroup$

What you want to maximize is the discounted stream of utility over time. To that end, you can set control variables over time (subject to some constraints, like technology, budget...). We are ultimately interested in the optimal setting of these variables (called the optimal policy).

Bellman's main insight is the principle of optimality (reminiscent of backward induction/subgame perfectness in game theory): suppose you choose some action today as part of an optimal policy, then the remaining action sequence must be part of an optimal policy from tomorrow on.

Before turning to the policy, we solve for the value function (this part is called the prediction problem). For a discrete action space, you could actually draw a complete decision tree and solve via backward induction. The more feasible, tractable, and elegant solution is to write the problem recursively with respect to the continuation value of the problem. Solving the Bellman Equation will give you that value function $V(\cdot)$, which is the optimal discounted value of the objective function, given the state you are in and an optimal policy from that state onwards. Note that the solution is thus a function, not a value.

From there, you can back out the sequence of actions (policy) that produces the optimal outcome. This part is called the control problem. Having solved the prediction problem, this is easy: the value function takes today's state and action as input and gives you the continuation value. So for each state, compute the continuation value for each possible action and pick the action that has the highest continuation value.

$\endgroup$
  • 1
    $\begingroup$ Hi Frederic, can you elaborate more by editing your answer on when you say "you can back out the sequence of actions that produces the optimal outcome". Thanks, $\endgroup$ – Frank Swanton Jun 14 at 0:18
  • 1
    $\begingroup$ Hi Frank, you are right, my first answer was a bit terse, I have expanded it a bit for clarity. I hope this helps. $\endgroup$ – Frederic Schneider Jun 14 at 11:17
  • $\begingroup$ Hi Fred, thanks for the elaboration. It helped a lot. I am reading few notes on this as it has been quite a long time the last time I dealt with the dynamic programming. When we use the term, the current value function $v(k)$ and the continuation value function, $v(k')$, what is the difference? Is it merely that we are putting the future capital stock as argument? Also, why is it the case that the Bellman equation has to hold for all feasible values of $k$? Why is this emphasized? $\endgroup$ – Frank Swanton Jun 18 at 8:08
0
$\begingroup$

Not sure what problem you are solving. But, the next state can be stochastic and the reward can be stochastic.

For example, In Inventory problem, your action is the number of items you purchase now. But next state can be calculated by current inventory(current state) + new purchase(action) - demand(unknown factor) therefore it will be stochastic.

In stochastic problem usually, you have Expectation in your objective function.

$\endgroup$
  • $\begingroup$ I am reviewing the deterministic optimal growth model, and this is not exactly the response I was looking for because stochastic growth model comes later. Thanks. $\endgroup$ – Frank Swanton Jun 14 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.