13
$\begingroup$

Consider the following differential equation \begin{align} \dot x(t)=f(x(t),u(t)) \end{align} where $x$ is the state and $u$ the control variable. The solution is given by \begin{align} x(t)=x_0 + \int^t_0f(x(s),u(s))ds. \end{align} where $x_0:=x(0)$ is the given inital state.

Now consider the following programm \begin{align} &V(x_0) := \max_u \int^\infty_0 e^{-\rho t}F(x(t),u(t))dt\\ s.t.~&\dot x(t)=f(x(t),u(t))\\ &x(0) = x_0 \end{align} where $\rho > 0$ denotes time preference, $V(\cdot)$ is the value and $F(\cdot)$ an objective function. A classical economic application is the Ramsey-Cass-Koopmans model of optimal growth. The Hamilton-Jacobi-Bellman equation is given by \begin{align} \rho V(x)=\max_u [F(x,u) + V'(x)f(x,u)],\quad \forall t\in[0,\infty). \end{align}

Say I've solved the HJB for $V$. The optimal control is then given by \begin{align} u^*=\arg\max_u [F(x,u) + V'(x)f(x,u)]. \end{align} I'll get optimal trajectories for the state and control $\{(x^*(t),u^*(t)):t\in[0,\infty)\}$.

The wiki article says

...but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum.

In Bertsekas (2005) Dynamic Programming and Optimal Control, Vol 1, 3rd ed., in Proposition 3.2.1 he states that solving for $V$ is the optimal cost-to-go function and the associated $u^*$ is optimal. However, he explicitly declares it as a sufficiency theorem.

Actually, I just want to make sure, that if I've solved the HJB and recoverd the associated state and control trajectories, that I don't have to be concerned with any additional optimality conditions.

Solution

I Attempt

I think I was able to derive necessary conditions from the maximum principle by the HJB equation itself.

Define the hamiltonian \begin{align} H(x,u,V'(x)) := F(x,u) + V'(x)f(x,u) \end{align}

then we have \begin{align} \rho V(x)=\max_u H(x,u,V'(x)) \end{align}

which is \begin{align} \rho V(x)= H(x,u^*,V'(x)). \end{align}

Define an arbitrary function $q:[0,\infty)\to\mathbb{R}$ with $q(0)=\lim_{t\to\infty} q(t)=0$. Now fix \begin{align} x = x^*+\varepsilon q \end{align}

where $\varepsilon\in\mathbb{R}$ is a parameter. Plug the term into the maximized hamiltonian which gives \begin{align} \rho V(x^*+\varepsilon q)= H(x^*+\varepsilon q,u^*,V'(x^*+\varepsilon q)). \end{align}

At $\varepsilon = 0$ we have the optimal solution. Thus differentite over $\varepsilon$ to get a first order condition \begin{align} \rho V'q = H_x q + H_{V'}V''q. \end{align}

Now define the adjoint variable with \begin{align} \lambda = V'(x). \end{align}

Differentiate over time \begin{align} \dot \lambda = V''\dot x. \end{align}

and note that \begin{align} H_{V'} = f(x,u) = \dot x. \end{align}

Plug everthing into the foc wich gives \begin{align} \rho \lambda = H_x + \dot \lambda. \end{align}

That's it pretty much. So solving the HJB is indeed necessary and sufficient (omitted here) for optimality. Someone should add it to wiki. Might save time for people thinking about such problems (won't be a lot I reckon).

However the transversality condition \begin{align} \lim_{t\to\infty} e^{-\rho t}\lambda(t) = 0 \end{align} is missing.

II Attempt

Define the payoff functional \begin{align} J(u):=\int^\infty_0 e^{-\rho t}F(x,u)dt \end{align}

Note that \begin{align} \int^\infty_0{e^{-\rho t}\lambda[f(x,u) - \dot x]dt} = 0 \end{align} by definition of $\dot x = f(x,u)$. Add the neutral Term to the payoff funtional \begin{align} J(u)&=\int^\infty_0 e^{-\rho t}[F(x,u)+\lambda f(x,u)]dt - \int^\infty_0{e^{-\rho t}\lambda\dot xdt}\\ &=\int^\infty_0 e^{-\rho t}H(x,u,\lambda) - \int^\infty_0{e^{-\rho t}\lambda\dot xdt} \end{align}

Integration by parts of the right term ond the rhs yields \begin{align} \int^\infty_0{e^{-\rho t}\lambda\dot xdt} = [e^{-\rho t}\lambda(t)x(t)]^\infty_0 - \int^\infty_0{e^{-\rho t}x(\dot \lambda-\rho\lambda)dt} \end{align}

Resubstitute that term \begin{align} J(u)=\int^\infty_0 e^{-\rho t}[H(x,u,\lambda) + x(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)x(t) + \lambda(0)x(0) \end{align}

Define \begin{align} x &= x^*+\varepsilon q\\ u &= u^*+\varepsilon p \end{align}

which gives \begin{align} J(\varepsilon)=\int^\infty_0 e^{-\rho t}[H(x^*+\varepsilon q,u^*+\varepsilon p,\lambda) + (x^*+\varepsilon q)(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)[x^*(t)+\varepsilon q(t)] + \lambda(0)x(0) \end{align}

FOC for maximum $J_\varepsilon = 0$ \begin{align} J_\varepsilon=\int^\infty_0 e^{-\rho t}[H_x q + H_u p + q(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)q(t) = 0 \end{align}

Since $q$ and $p$ are unconstrained we must have \begin{align} H_u &= 0\\ H_x &= \rho\lambda - \dot \lambda\\ \lim_{t\to\infty}e^{-\rho t}\lambda(t) &= 0 \end{align}

$\endgroup$
  • $\begingroup$ have you identified the necessary and sufficient conditions yet? $\endgroup$ – Jamzy Jul 8 '15 at 22:47
  • $\begingroup$ In what economic context does this come up? $\endgroup$ – Stan Shunpike Jul 11 '15 at 5:04
  • $\begingroup$ Ramsey model for instance cer.ethz.ch/resec/people/tsteger/Ramsey_Model.pdf $\endgroup$ – clueless Jul 11 '15 at 8:12
  • 1
    $\begingroup$ I think that this thread is better suited for math.stackexchange.com since it is not really linked to econ. A mod may transfer it. $\endgroup$ – clueless Jul 14 '15 at 13:22
  • $\begingroup$ I am not sure what is asked here: if per Bertsekas solving HJB is sufficient, then you do not have to "worry about additional optimality conditions". The "sufficient only" against "necessary and sufficient" would arise in case HJB was not solved -in which case one would say "this does not mean that there is no solution". By the way, your Attempts I and II are valuable content here - the first showing a link bewteen HJB and Optimal Control, the second showing how the Optimal Control FOCs can be derived. $\endgroup$ – Alecos Papadopoulos Jul 29 '15 at 10:00
1
$\begingroup$

(This perhaps should be considered a comment.)

If you have solved the HJB equation, it is sufficient to get the optimal solution. So you do not "have to be concerned with any other optimality conditions," which I believe appears to answer your question.

It appears that you are concerned about the "necessary" component of the theorem. The necessity side of the statement is as follows: if there is an optimal solution, there must exist a solution to the HJB equation.

I have not worked with this particular problem, but the answer in general is that we do not expect to have a differentiable function V. Therefore we do not have a solution to the equation as it is stated. Instead, we need to look at generalised derivatives, and convert the HJB equation into an inequality. In which case, you may get a "viscosity solution." If we extend to use generalised derivatives, it may be possible to prove that a such a solution always exists. Glancing at your proofs, they will not help on the necessity conditions, as you are assuming differentiability.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.