There are two agents $i=1,2$. The state $k$ is governed by $\tau_i\in[0,1]$ where \begin{align} \dot{k} = f(k,\tau_1,\tau_2). \end{align}

Define the value function of player $i$ by \begin{align} v_i(k) := \sup_{\tau_i}\int^\infty_0{e^{-\rho t}F(k,\tau_i,\sigma_{-i})dt}. \end{align} where $\sigma_{-i}(k)$ is the policy function or a markovian stratgey profile of player $-i$. And $\rho>0$ is the time preference rate.

The Hamilton-Jacobi-Bellman equation reads \begin{align} \rho v_i(k) = \sup_{\tau_i}\{F(k,\tau_i,\sigma_{-i}) + v'(k)f(k,\tau_i,\sigma_{-i})\}. \end{align}

Now assume I approximated a function $\hat{v}_i$ by value function iteration. How do I know that it's the true value function?

  • $\begingroup$ Check the distance with the last iteration. If you like it, you can stop. If it's too big, keep iterating. $\endgroup$
    – VicAche
    Commented Jun 5, 2015 at 9:17
  • $\begingroup$ first of all you can solve an analytical example and then test your algorithm against it. Second, the value function is a fixed point, so you can google for some tests of convergence to a fixed point, at least one heurisitic rules comes to mind, the distance between i and i+1 is decreasing , strict concavity ensures global optimality $\endgroup$
    – user157623
    Commented Jun 8, 2015 at 12:20

1 Answer 1


The first panel shows the final value function after iterating 80k (!). Should probably try an implicit method. The second panel shows the evolution from frist guess to final result $v^0\to v^j$, $j=80k$. Third panel is the distance between the value functions and fourth shows the error.

Note that the value function is convex-concave. This is due to the fact that for the limit case the law of motion tends to infinty $\lim_{k\to 0} \dot{k} = \infty$.

By the maximum principle optimality requires that the Hamiltonian \begin{align} H_1(k,\tau_1,\tau_2^*,\lambda_1) := F(k,\tau_1,\tau_2^*) + \lambda_1 f(k,\tau_1,\tau_2^*) \end{align} is concave in $(k,\tau_1,\tau_2)$ (sufficient condition). Now we have
\begin{align} \rho v(k) = \max_{\tau_1} H_1(k,\tau_1,\tau_2^*,\lambda_1). \end{align}

  • I was wondering if the value function must be (strictly) concave to ensure optimality?

Any reference is appreciated.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.