# References to learn continuous-time dynamic programming

Does anyone know of good references to learn continuous-time dynamic programming? The references don't have to be books. They could be links to online resources as well. Links to clear, concise discussions of even just the basics would be helpful.

• deterministic, stochastic, or both? depends quite a bit on which. – nominally rigid Jan 5 '15 at 23:08
• Sorry. Both would be helpful, but I actually do have stochastic in mind. Thanks! – jmbejara Jan 5 '15 at 23:09
• Just thought I'd mention that if we are going to do these types of questions, we should keep with the decided convention of providing 1 recommendation per answer. – cc7768 Jan 9 '15 at 18:41

For continuous-time stochastic dynamic programming, the small, nontechnical Art of Smooth Pasting by Dixit is a wonderful option. It does a very effective job of conveying the basic intuition.

Stokey's more recent The Economics of Inaction is also decent, but for a practical-minded person it probably underperforms Dixit - its much greater length and somewhat heavier notation do not yield commensurate rewards.

If the underlying stochastic processes are not Itō diffusions, then I'm not sure what the best reference is. The most common such case I've seen (and that I use myself) is the case of discretely many exogenous states, where if we are currently in state $s$ there is some constant hazard rate $\lambda_{s,s'}$ of a switch to state $s'$. Fortunately, this is a pretty simple case in practice: one can just modify the HJB equation to account for the flow probability of switching from $V(\cdot,s)$ to $V(\cdot,s')$. (You can see this, for instance, in equations (1)-(5) in this Acemoglu and Akcigit paper. Conceptually it's no different from setting up the HJB equation when we have an Itō diffusion as the driving process, except that it's simpler because we just get a system of linear equations and we don't need to think about Itō's lemma, etc.)

Of course, maybe there is a good textbook reference for this too - but unlike the potentially much more complicated cases involving stochastic calculus, this is straightforward enough that a text has never seemed necessary to me.

• The Economics of Inaction looks great. I looked a little at it. Given your description of The Art of Smooth Pasting, I'd really like to check it out, but it seems a little hard to get a hold of. Thanks for the recommendations! – jmbejara Jan 7 '15 at 5:27

Dynamic Programming & Optimal Control by Bertsekas

Introduction to Modern Economic Growth by Acemoglu

The Acemoglu book, even though it specializes in growth theory, does a very good job presenting continuous time dynamic programming.

I think Kamien and Schwartz's Dynamic Optimization: The Calculus of Variations and Optimal Control in Economics and Management is pretty well known.

• It is a very good but note that the book is not really an introduction book for dynamic optimization. – optimal control Sep 24 '15 at 9:09

Controlled Markov Processes and Viscosity Solutions by Fleming and Soner includes a number of applications to Finance and Differential Games.

A really nice methodology for approximating the HJB is the upwind scheme, which I learnt quite quickly using Ben Moll et al's notes and codes

The examples are continuous time versions of familiar heterogenous agents economies models such as Hugget and Aiyagari.

Applied Intertemporal Optimization by Klaus Wälde is a very very nice book, even for those who are not really familiar with mathematics.

The book treats deterministic and stochastic models, both in discrete and continuous time.

I would really say for this book "Dynamic Optimization for dummies". I was not familiar at all with dynamic optimization but this book let me to get through.

• This book looks fantastic. I love the organization. The tables at the beginning really illustrate how thorough it is. The book covers most cases of interest and provides many examples. (Apologies for the 4 year delay in responding... :) ) – jmbejara Mar 29 at 16:57
• Happy that it helps! I learned so much from this book :) – optimal control Mar 29 at 17:16