Dynamic optimisation

Question

Consider a simple dynamic consumption-saving problem. A solution can be characterised using a Lagrangian approach that generates a set of first order conditions and some boundary conditions.

An alternative approach is to set use a Bellman equation. What I don't understand is how come a Bellman equation approach does not seem to use information regarding the boundary conditions in the solution methodology? That is, a Bellman equation does have a tradeoff between time period t and t+1 but not some overall budget constraint does not seem to be represented.

Answer 1

The Dynamic Programming ("Bellman' Equation") formulation incorporates the terminal boundary condition ("transversality conditions") needed in case we use the Lagrangian/Euler equation formulation. The initial conditions are still needed in both approaches.

To see why, consider the problem

$$\text{max}_{\{x_t\}} \sum_{t=0}^{\infty} \beta^t U(c_t)$$

$$s.t.\;\;\; a_{t+1}= (1-r)a_t + w - c_t, \;\;\; a_0 \; \text{given}$$

Solving the budget constraint for $c_t$ and inserting into the objective function, we have compactly

$$\text{max}_{\{a_{t}\}} \sum_{t=0}^{\infty} \beta^t U(a_t, a_{t+1}),\;\;\;\; a_0 \; \text{given}$$

where we treat the stock variable next period as our decision variable. Now for the optimal sequence, $a^*_{t+1}$ must solve the following problem:

$$\max_y \big[U(a^*_t, y) + \beta U(y, a^*_{t+2})\big],\;\;\; y\;\; \text{feasible given}\;\; a^*_t$$

In the words of Stokey and Lucas "a feasible variation on the sequence $\{a^*_t\}$ cannot lead to an improvement on an optimal policy".

Then we get the first order conditions (differentiating with respect to $y$ essentially)

$$U_2(a^*_t, a^*_{t+1}) + \beta U_1(a^*_{t+1}, a^*_{t+2}) =0$$

where the subscripts $1,2$ denote partial derivatives. This is a second-order difference equation, and so it requires two boundary conditions. We have already one of them, the given initial value of wealth $a_0$. We need one more, hence the terminal, "transversality" condition.

In contrast, using the Dynamic Programming approach we arrive at a first-order difference equation, and we do not need an additional boundary condition.

For the full exposition, see Stokey & Lucas p. 97-100.

Answer 2

I'm unsure exactly where your struggle is, in general. To try and address your issues, what kind of boundary conditions might we desire in dynamic problems?

Consider the two period consumption-savings problem where we have

$$c_1 + s_1 = y_1$$ $$c_2 = y_2 + (1+R)s_1$$

$y_1, y_2$ being endowments in each period, $s_1$ as savings, $R$ for interest, and $c_1, c_2$ for consumption. Given some $U(c_1, c_2)$, we want to solve for optimal consumption in each period. We can consolidate the budget as:

$$c_2 = y_2 + (1 + R)(y_1 - c_1)$$ $$\implies c_1 + \frac{c_2}{1 + R} = y_1 + \frac{y_2}{1 + R}$$

and express the Lagrangian problem:

$$\mathcal{L} = U(c_1, c_2) - \lambda(c_1 - \frac{c_2}{1 + R} - y_1 - \frac{y_2}{1 + R})$$

Solving for FOCs gets us:

$$U_{c_1} - \lambda = 0$$ $$U_{c_2} - \frac{\lambda}{1 + R} = 0$$ $$\implies \frac{U_{c_1}}{U_{c_2}} = 1 + R$$

But you can also express this problem as a Bellman, and still keep that original consolidated budget constraint.

$$\max_{c_1, c_2, y_1, y_2} U(c_1, c_2) \quad \text{s.t.} \ c_1 + \frac{c_2}{1 + R} = y_1 + \frac{y_2}{1 + R}$$

where you have a value function for $y_1$ (yes, $y_1$ is still given)

$$V(y_1) = \max_{c_1, c_2, y_2} \ U(c_1, c_2) \quad \text{s.t.} \ y_1 = c_1 + \frac{c_2}{1 + R} - \frac{y_2}{1 + R}$$

To make this problem less trivial for a Bellman, you can have a constraint where $c$ determines $y$ somehow instead. But otherwise you can still incorporate the budget constraint into your problem here, albeit it might look pretty weird.