What is unknown in Bellman Equation?

Question

\begin{align} V(W)=\max\limits_{W'\in[0,W]}\qquad& u(W-W')+\beta V(W')\qquad\forall W \end{align}

$\textbf{My Question}$: Why is the unknown in the Bellman equation $V(W)$ itself? Isn't the choice variable tomorrow's state, $W'$?

Answer 1

The original problem was probably of the form $$\max_{\{W_t\}_{t=1}^\infty}\sum_{t=0}^\infty \beta^t u(W_t-W_{t+1}),$$ $$\mbox{s.t. } W_{t+1}\in[0,W_t] \ \forall \ t, ~~ W_0 \mbox{ given}$$ When formulated that way there is no need to solve for any function $V$, but the infinite elements of the sequence ${\{W_t\}}_{t=1}^\infty$ are all choice variables. Bellman's main insight is that (under some conditions) the problem can be simplified. Instead of solving for an infinite number of variables, you could solve for a single "policy function" that tells you what to do each period as a function of the current state variables, and a "value function" that summarizes what is the value (in terms of today) of inducing each possible state in the future. Even if there is no uncertainty, it is clear that this value function is an endogenous object, since it must somehow capture the future stream of utility as a single function of the future state. In your case, that function is $V(W')$. This is why the Bellman equation: $$V(W)=\max_{W'\in[0,W]} u(W-W')+\beta V(W')$$ has two properties:

1) it lacks the subscripts on the $W$'s, this reflects that you will solve for $W'$ as a function of $W$, i.e. the policy function.

2) has a new object, $V$, that was not in the original problem and it is a function of $W$ (or $W'$) and just by eyeballing the two problems you can see that for them to be equivalent $V$ must be the discounted sum of future payoffs assuming future actions are also optimal. So it is not a choice variable, but definitely, an endogenous object that you need to find when solving the problem.

You can see that the problem of optimally choosing infinitely many objects is transformed into one of choosing a finite number (2) of functions, which are infinite-dimensional objects. Brilliant!

Answer 2

What you want to maximize is the discounted stream of utility over time. To that end, you can set control variables over time (subject to some constraints, like technology, budget...). We are ultimately interested in the optimal setting of these variables (called the optimal policy).

Bellman's main insight is the principle of optimality (reminiscent of backward induction/subgame perfectness in game theory): suppose you choose some action today as part of an optimal policy, then the remaining action sequence must be part of an optimal policy from tomorrow on.

Before turning to the policy, we solve for the value function (this part is called the prediction problem). For a discrete action space, you could actually draw a complete decision tree and solve via backward induction. The more feasible, tractable, and elegant solution is to write the problem recursively with respect to the continuation value of the problem. Solving the Bellman Equation will give you that value function $V(\cdot)$, which is the optimal discounted value of the objective function, given the state you are in and an optimal policy from that state onwards. Note that the solution is thus a function, not a value.

From there, you can back out the sequence of actions (policy) that produces the optimal outcome. This part is called the control problem. Having solved the prediction problem, this is easy: the value function takes today's state and action as input and gives you the continuation value. So for each state, compute the continuation value for each possible action and pick the action that has the highest continuation value.

Answer 3

Not sure what problem you are solving. But, the next state can be stochastic and the reward can be stochastic.

For example, In Inventory problem, your action is the number of items you purchase now. But next state can be calculated by current inventory(current state) + new purchase(action) - demand(unknown factor) therefore it will be stochastic.

In stochastic problem usually, you have Expectation in your objective function.