# Solving the Hamilton-Jacobi-Bellman equation; necessary and sufficient for optimality?

Consider the following differential equation \begin{align} \dot x(t)=f(x(t),u(t)) \end{align} where $x$ is the state and $u$ the control variable. The solution is given by \begin{align} x(t)=x_0 + \int^t_0f(x(s),u(s))ds. \end{align} where $x_0:=x(0)$ is the given inital state.

Now consider the following programm \begin{align} &V(x_0) := \max_u \int^\infty_0 e^{-\rho t}F(x(t),u(t))dt\\ s.t.~&\dot x(t)=f(x(t),u(t))\\ &x(0) = x_0 \end{align} where $\rho > 0$ denotes time preference, $V(\cdot)$ is the value and $F(\cdot)$ an objective function. A classical economic application is the Ramsey-Cass-Koopmans model of optimal growth. The Hamilton-Jacobi-Bellman equation is given by \begin{align} \rho V(x)=\max_u [F(x,u) + V'(x)f(x,u)],\quad \forall t\in[0,\infty). \end{align}

Say I've solved the HJB for $V$. The optimal control is then given by \begin{align} u^*=\arg\max_u [F(x,u) + V'(x)f(x,u)]. \end{align} I'll get optimal trajectories for the state and control $\{(x^*(t),u^*(t)):t\in[0,\infty)\}$.

The wiki article says

...but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum.

In Bertsekas (2005) Dynamic Programming and Optimal Control, Vol 1, 3rd ed., in Proposition 3.2.1 he states that solving for $V$ is the optimal cost-to-go function and the associated $u^*$ is optimal. However, he explicitly declares it as a sufficiency theorem.

Actually, I just want to make sure, that if I've solved the HJB and recoverd the associated state and control trajectories, that I don't have to be concerned with any additional optimality conditions.

## Solution

I Attempt

I think I was able to derive necessary conditions from the maximum principle by the HJB equation itself.

Define the hamiltonian \begin{align} H(x,u,V'(x)) := F(x,u) + V'(x)f(x,u) \end{align}

then we have \begin{align} \rho V(x)=\max_u H(x,u,V'(x)) \end{align}

which is \begin{align} \rho V(x)= H(x,u^*,V'(x)). \end{align}

Define an arbitrary function $q:[0,\infty)\to\mathbb{R}$ with $q(0)=\lim_{t\to\infty} q(t)=0$. Now fix \begin{align} x = x^*+\varepsilon q \end{align}

where $\varepsilon\in\mathbb{R}$ is a parameter. Plug the term into the maximized hamiltonian which gives \begin{align} \rho V(x^*+\varepsilon q)= H(x^*+\varepsilon q,u^*,V'(x^*+\varepsilon q)). \end{align}

At $\varepsilon = 0$ we have the optimal solution. Thus differentite over $\varepsilon$ to get a first order condition \begin{align} \rho V'q = H_x q + H_{V'}V''q. \end{align}

Now define the adjoint variable with \begin{align} \lambda = V'(x). \end{align}

Differentiate over time \begin{align} \dot \lambda = V''\dot x. \end{align}

and note that \begin{align} H_{V'} = f(x,u) = \dot x. \end{align}

Plug everthing into the foc wich gives \begin{align} \rho \lambda = H_x + \dot \lambda. \end{align}

That's it pretty much. So solving the HJB is indeed necessary and sufficient (omitted here) for optimality. Someone should add it to wiki. Might save time for people thinking about such problems (won't be a lot I reckon).

However the transversality condition \begin{align} \lim_{t\to\infty} e^{-\rho t}\lambda(t) = 0 \end{align} is missing.

II Attempt

Define the payoff functional \begin{align} J(u):=\int^\infty_0 e^{-\rho t}F(x,u)dt \end{align}

Note that \begin{align} \int^\infty_0{e^{-\rho t}\lambda[f(x,u) - \dot x]dt} = 0 \end{align} by definition of $\dot x = f(x,u)$. Add the neutral Term to the payoff funtional \begin{align} J(u)&=\int^\infty_0 e^{-\rho t}[F(x,u)+\lambda f(x,u)]dt - \int^\infty_0{e^{-\rho t}\lambda\dot xdt}\\ &=\int^\infty_0 e^{-\rho t}H(x,u,\lambda) - \int^\infty_0{e^{-\rho t}\lambda\dot xdt} \end{align}

Integration by parts of the right term ond the rhs yields \begin{align} \int^\infty_0{e^{-\rho t}\lambda\dot xdt} = [e^{-\rho t}\lambda(t)x(t)]^\infty_0 - \int^\infty_0{e^{-\rho t}x(\dot \lambda-\rho\lambda)dt} \end{align}

Resubstitute that term \begin{align} J(u)=\int^\infty_0 e^{-\rho t}[H(x,u,\lambda) + x(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)x(t) + \lambda(0)x(0) \end{align}

Define \begin{align} x &= x^*+\varepsilon q\\ u &= u^*+\varepsilon p \end{align}

which gives \begin{align} J(\varepsilon)=\int^\infty_0 e^{-\rho t}[H(x^*+\varepsilon q,u^*+\varepsilon p,\lambda) + (x^*+\varepsilon q)(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)[x^*(t)+\varepsilon q(t)] + \lambda(0)x(0) \end{align}

FOC for maximum $J_\varepsilon = 0$ \begin{align} J_\varepsilon=\int^\infty_0 e^{-\rho t}[H_x q + H_u p + q(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)q(t) = 0 \end{align}

Since $q$ and $p$ are unconstrained we must have \begin{align} H_u &= 0\\ H_x &= \rho\lambda - \dot \lambda\\ \lim_{t\to\infty}e^{-\rho t}\lambda(t) &= 0 \end{align}

• have you identified the necessary and sufficient conditions yet? – Jamzy Jul 8 '15 at 22:47
• In what economic context does this come up? – Stan Shunpike Jul 11 '15 at 5:04
• Ramsey model for instance cer.ethz.ch/resec/people/tsteger/Ramsey_Model.pdf – clueless Jul 11 '15 at 8:12
• I think that this thread is better suited for math.stackexchange.com since it is not really linked to econ. A mod may transfer it. – clueless Jul 14 '15 at 13:22
• I am not sure what is asked here: if per Bertsekas solving HJB is sufficient, then you do not have to "worry about additional optimality conditions". The "sufficient only" against "necessary and sufficient" would arise in case HJB was not solved -in which case one would say "this does not mean that there is no solution". By the way, your Attempts I and II are valuable content here - the first showing a link bewteen HJB and Optimal Control, the second showing how the Optimal Control FOCs can be derived. – Alecos Papadopoulos Jul 29 '15 at 10:00