Simulating a Hamilton-Jacobi-Bellman

Say I have solved an HJB of the form:

$\rho V(k) = \max_c g(c) + V'(k)(z - c)$

I have calibrated $\rho$ to monthly parameters. I would like to simulate the development of $k$. I start with $k(0)$. However, unlike in discrete-time, I'm not sure what happens next.

Is $k(1) = k(0) + (z-c(0))$, with $k(1)$ being the value of $k$ for the next month? Or do I need to somehow integrate along the lines of $k(1) = \int_0^1 (z-c(t))dt + k(0)$? There's many references on solving these things, but I couldn't find any references on simulating it.

• I believe it would be helpful if you could include in the question's title the full name, prior to use the initials HJB. "Hamilton-Jacobi-Bellman" may not come immediately to mind, to those browsing the list of question titles. Nov 19 '14 at 18:16
• I nearly always work in discrete time for macro, for this reason. That said, I've written simulations for continuous-time, non-macro econ models. When I've done that, the approach was to decide on a $\Delta$t step size, change all equations to difference equations (decide some timing issues), and then simulate. Unfortunately I have no immediate public examples of this for you -- and unsure it actually answers your question (hence the comment instead of an answer). Nov 20 '14 at 16:35
• @Foobar: Have you tried the integration method you suggested? How did that go if you did? Nov 20 '14 at 16:36
• Well, it succeeds in giving me a number. It's hard to actually falsify or validate the result. Nov 20 '14 at 16:39

1. Solve the HJB equation $\rho V(k) = \max_c g(c)+V'(k)(z-c)$. (You're already done this.)
2. Obtain the policy function $c(k)=\text{argmax}_c g(c)+V'(k)(z-c)$. (You presumably already did this while solving the HJB.)
3. Substitute $c(k)$ into the law of motion for $k$ and numerically solve that differential equation for $k(t)$:$$\dot{k}=z-c(k)$$

It sounds like you already had (1) and (2) and were just unsure about what exactly to do in (3). If you're working in a deterministic environment (and from the HJB it looks like you are), no discretization on your part is necessary: you can just use a canned ODE solver from Matlab or any other software package to numerically solve the differential equation above for $k(t)$.

To be a little more specific:

• If the HJB is simple and you have an analytical solution for $V(k)$, and thus can easily compute $c(k)$ as well, that's great. You can just plug $c(k)$ into the law of motion for $k$ and tell your software package of choice to solve it and obtain the trajectory of $k(t)$.
• If you had to numerically solve the HJB, then that's great because it means that you probably have already used some kind of numerical ODE solution technique and will be familiar with it. On the minus side, it means that when you initially obtain $V(k)$ and $c(k)$, you'll only get it for a grid of $k$, rather than having some kind of formula you can apply everywhere - which is a problem, because you'll need to be able to determine $c(k)$ at any point $k$ in a continuum if you're going to numerically solve the $\dot{k}$ differential equation for $k(t)$. The simplest way around this, to my knowledge, is to fit a spline to (or use some other approximation method on) the discrete grid of $(k_{grid},c(k_{grid}))$ values that you have. Then you can obtain an approximate value of $c(k)$ anywhere that's covered by the spline, and it will be extremely accurate assuming that your ODE is well-behaved and you have a dense grid.

Things get more complicated in a stochastic environment, but your case looks deterministic. (And to be clear, I'm just a dabbler here: numerical methods are not my area of expertise. But this is what I know.)

You work in discrete time, see for example this resource.

Or this (incomplete) paper, where the same author participates. In the beginning the paper talks about "coarse discrete methods" and may give the impression that it contains some continuous time simulation method. It really doesn't -see the description of their algorithm in the Appendix, page 30.

Also this MPhil dissertation, which, being a graduate thesis, describes the steps in some detail, and with a wealth of referenced literature.

• Both Ben's notes and his paper don't really mention how to simulate them, only how to solve them - and that's not my problem.. Nov 19 '14 at 19:56

Since Ben's notes were already mentioned I'm not quite sure why you don't iterate over time using a forward approximation as he suggests \begin{align} \dot k(t) :\approx \frac{k(t+1) - k(t)}{\Delta t} \Leftrightarrow k(t+1) \approx \Delta t \dot k(t) + k(t) \end{align}

which gives \begin{align} k(1) \approx \Delta t (z-c(0)) + k(0) \end{align}

• You can also look at the approximations made with finite difference equation. It is a good method do discretize a continous time model. Sep 10 '15 at 18:16

Once you have solved the equation $\rho V(k) = \max_c g(c) + V'(k)(z - c)$ to find the function $V(k)$ you should be able to solve for what is called the policy function. The function $V(k)$ is the value function. Knowing $V(k)$ should then give you the information that you need to solve for the policy, say $c(k,t)$. This will then tell you how $k$ will evolve over time.

• I know $c(k,t)$, my problem is how to understand it. How exactly do I get $k(1)$, given that I know the policy function and $k(0)$? Nov 19 '14 at 22:06