4
$\begingroup$

When solving (numerically, by value function iteration) a dynamic programming problem in discrete time, such as

$$V_1(a) = \max_{c} \ u(c) + \dfrac{1}{1+\rho}V_0(a')$$

we maximize with respect to the control variable and get a first order condition that we then plug back into the functional equation shown above. The result of this step, $V(a)_1$, will then be used on the RHS of a second iteration

$$V_2(a) = \max_{c} \ u(c) + \dfrac{1}{1+\rho}V_1(a')$$

and we repeat this process until $V(a)_n-V(a)_{n+1}<\epsilon$.

My question is how does the update of the value function work in continuous time? I have been working on a paper that uses continuous time dynamic programming, so the Bellman equation looks as follows

$$\rho V_n(a) = \max_{c} \ u(c) + \dfrac{\partial V_n(a)}{\partial a}da_t \quad (*)$$

where the transition equation is represented by $da_t$. From what I have seen, the update of the value function is done by calculating $\Delta$:

$$ \Delta = \ u(c(a^*)) + \dfrac{\partial V_n(a)}{\partial a}da_t(a^*) - \rho V_n(a)$$

where $u(c(a^*))$ and $da_t(a^*)$ represent the control and transition equation as functions of the optimal policy. That is, we maximize the RHS as in the previous example (the discrete time case), but then we subtract $\rho V(a)$ from both sides. Then updating the value function is done as follows:

$$V_{n+1}(a) = V_n(a) + \Delta$$

How can this be so? I would have thought that I would just use the maximised RHS of (*) and plug back in a new iteration. How come the other method is the correct one?

Improve this question
$\endgroup$
4
  • $\begingroup$ Are you asking how this problem is solved numerically? Your notation is confusing and it's difficult to tell what you're asking. $\endgroup$
    – NickJ
    Commented Apr 21, 2016 at 2:08
  • $\begingroup$ Hi Nick, yes I am asking how it is solved numerically. Value function iteration usually maxes the RHS of the functional equation, obtains a policy rule from the FOC and then plugs it back in on the RHS. This step is done over and over until the value function converges. This is different from the method shown in the last equation, you may see that the value function iteration is done differently: we still maximize the RHS but then use the additive method where we add delta to the value function I started with. $\endgroup$
    – Sophie
    Commented Apr 21, 2016 at 2:22
  • 1
    $\begingroup$ Usually (always?) when doing value function iteration, we discretize the state space and maximize numerically, no first order conditions are taken. Can I re-phrase your question as the following? "I know how to use value function iteration to find a numerical approximation to the Bellman equation for a discrete time dynamic programming problem, but how do I numerically approximate the solution when time is continuous?" $\endgroup$
    – NickJ
    Commented Apr 21, 2016 at 13:14
  • $\begingroup$ Check out Ben Moll's stuff. Very transparent with codes also. princeton.edu/~moll/HACTproject/HACT_Additional_Codes.pdf $\endgroup$
    – clueless
    Commented Apr 22, 2016 at 10:55

1 Answer 1

Reset to default
2
$\begingroup$

You iterate towards a fixed point, so you want to reach a situation where plugging in your current iterated value produces itself. Now using your notation, we are told that we should calculate

$$V_{n+1}(a) = V_n(a) + \Delta$$

where

$$\Delta = \ u(c(a^*)) + \dfrac{\partial V_n(a)}{\partial a}da_t(a^*) - \rho V_n(a)$$

Insert the second into the first to see what the iteration rule is:

$$V_{n+1}(a) = V_n(a) + \ u(c(a^*)) + \dfrac{\partial V_n(a)}{\partial a}da_t(a^*) - \rho V_n(a)$$

When you reach a point where

$$V_{n+1}(a) = V_n(a) $$

(or $\epsilon$-so)

It will mean

$$\rho V_n(a)= \ u(c(a^*)) + \dfrac{\partial V_n(a)}{\partial a}da_t(a^*) $$

which is what you have to satisfy.

Some stars etc may have to be adjusted in the above, for a fully consistent notation.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.