Return to Question

Consider the following differential equation \begin{align} \dot x(t)=f(x(t),u(t)) \end{align} where $x$ is the state and $u$ the control variable. The solution is given by \begin{align} x(t)=x_0 + \int^t_0f(x(s),u(s))ds. \end{align} where $x_0:=x(0)$ is the given inital state.

Now consider the following programm \begin{align} &V(x_0) := \max_u \int^\infty_0 e^{-\rho t}F(x(t),u(t))dt\\ s.t.~&\dot x(t)=f(x(t),u(t))\\ &x(0) = x_0 \end{align} where $\rho > 0$ denotes time preference, $V(\cdot)$ is the value and $F(\cdot)$ an objective function. A classical economic application is the Ramsey-Cass-Koopmans model of optimal growth. The Hamilton-Jacobi-Bellman equation is given by \begin{align} \rho V(x)=\max_u [F(x,u) + V'(x)f(x,u)],\quad \forall t\in[0,\infty). \end{align}

Say I've solved the HJB for $V$. The optimal control is then given by \begin{align} u^*=\arg\max_u [F(x,u) + V'(x)f(x,u)]. \end{align} I'll get optimal trajectories for the state and control $\{(x^*(t),u^*(t)):t\in[0,\infty)\}$.

The wiki article says

...but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum.

In Bertsekas (2005) Dynamic Programming and Optimal Control, Vol 1, 3rd ed., in Proposition 3.2.1 he states that solving for $V$ is the optimal cost-to-go function and the associated $u^*$ is optimal. However, he explicitly declares it as a sufficiency theorem.

Actually, I just want to make sure, that if I've solved the HJB and recoverd the associated state and control trajectories, that I don't have to be concerned with any additional optimality conditions.

Addendum: Solution

I think I was able to derive necessary conditions from the maximum principle by the HJB equation itself.

Define the hamiltonian \begin{align} H(x,u,V'(x)) := F(x,u) + V'(x)f(x,u) \end{align}

then we have \begin{align} \rho V(x)=\max_u H(x,u,V'(x)) \end{align}

which is \begin{align} \rho V(x)= H(x,u^*,V'(x)). \end{align}

Define an arbitrary function $q:[0,\infty)\to\mathbb{R}$ with $q(0)=\lim_{t\to\infty} q(t)=0$. Now fix \begin{align} x = x^*+\varepsilon q \end{align}

where $\varepsilon\in\mathbb{R}$ is a parameter. Plug the term into the maximized hamiltonian which gives \begin{align} \rho V(x^*+\varepsilon q)= H(x^*+\varepsilon q,u^*,V'(x^*+\varepsilon q)). \end{align}

At $\varepsilon = 0$ we have the optimal solution. Thus differentite over $\varepsilon$ to get a first order condition \begin{align} \rho V'q = H_x q + H_{V'}V''q. \end{align}

Now define the adjoint variable with \begin{align} \lambda = V'(x). \end{align}

Differentiate over time \begin{align} \dot \lambda = V''\dot x. \end{align}

and note that \begin{align} H_{V'} = f(x,u) = \dot x. \end{align}

Plug everthing into the foc wich gives \begin{align} \rho \lambda = H_x + \dot \lambda. \end{align}

That's it pretty much. So solving the HJB is indeed necessary and sufficient (omitted here) for optimality. Someone should add it to wiki. Might save time for people thinking about such problems (won't be a lot I reckon).

However the transversality condition \begin{align} \lim_{t\to\infty} e^{-\rho t}\lambda(t) = 0 \end{align} is missing.

Consider the following differential equation \begin{align} \dot x(t)=f(x(t),u(t)) \end{align} where $x$ is the state and $u$ the control variable. The solution is given by \begin{align} x(t)=x_0 + \int^t_0f(x(s),u(s))ds. \end{align} where $x_0:=x(0)$ is the given inital state.

Now consider the following programm \begin{align} &V(x_0) := \max_u \int^\infty_0 e^{-\rho t}F(x(t),u(t))dt\\ s.t.~&\dot x(t)=f(x(t),u(t))\\ &x(0) = x_0 \end{align} where $\rho > 0$ denotes time preference, $V(\cdot)$ is the value and $F(\cdot)$ an objective function. A classical economic application is the Ramsey-Cass-Koopmans model of optimal growth. The Hamilton-Jacobi-Bellman equation is given by \begin{align} \rho V(x)=\max_u [F(x,u) + V'(x)f(x,u)],\quad \forall t\in[0,\infty). \end{align}

Say I've solved the HJB for $V$. The optimal control is then given by \begin{align} u^*=\arg\max_u [F(x,u) + V'(x)f(x,u)]. \end{align} I'll get optimal trajectories for the state and control $\{(x^*(t),u^*(t)):t\in[0,\infty)\}$.

The wiki article says

...but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum.

In Bertsekas (2005) Dynamic Programming and Optimal Control, Vol 1, 3rd ed., in Proposition 3.2.1 he states that solving for $V$ is the optimal cost-to-go function and the associated $u^*$ is optimal. However, he explicitly declares it as a sufficiency theorem.

Actually, I just want to make sure, that if I've solved the HJB and recoverd the associated state and control trajectories, that I don't have to be concerned with any additional optimality conditions.

Solution

I Attempt

I think I was able to derive necessary conditions from the maximum principle by the HJB equation itself.

Define the hamiltonian \begin{align} H(x,u,V'(x)) := F(x,u) + V'(x)f(x,u) \end{align}

then we have \begin{align} \rho V(x)=\max_u H(x,u,V'(x)) \end{align}

which is \begin{align} \rho V(x)= H(x,u^*,V'(x)). \end{align}

Define an arbitrary function $q:[0,\infty)\to\mathbb{R}$ with $q(0)=\lim_{t\to\infty} q(t)=0$. Now fix \begin{align} x = x^*+\varepsilon q \end{align}

where $\varepsilon\in\mathbb{R}$ is a parameter. Plug the term into the maximized hamiltonian which gives \begin{align} \rho V(x^*+\varepsilon q)= H(x^*+\varepsilon q,u^*,V'(x^*+\varepsilon q)). \end{align}

At $\varepsilon = 0$ we have the optimal solution. Thus differentite over $\varepsilon$ to get a first order condition \begin{align} \rho V'q = H_x q + H_{V'}V''q. \end{align}

Now define the adjoint variable with \begin{align} \lambda = V'(x). \end{align}

Differentiate over time \begin{align} \dot \lambda = V''\dot x. \end{align}

and note that \begin{align} H_{V'} = f(x,u) = \dot x. \end{align}

Plug everthing into the foc wich gives \begin{align} \rho \lambda = H_x + \dot \lambda. \end{align}

That's it pretty much. So solving the HJB is indeed necessary and sufficient (omitted here) for optimality. Someone should add it to wiki. Might save time for people thinking about such problems (won't be a lot I reckon).

However the transversality condition \begin{align} \lim_{t\to\infty} e^{-\rho t}\lambda(t) = 0 \end{align} is missing.

II Attempt

Define the payoff functional \begin{align} J(u):=\int^\infty_0 e^{-\rho t}F(x,u)dt \end{align}

Note that \begin{align} \int^\infty_0{e^{-\rho t}\lambda[f(x,u) - \dot x]dt} = 0 \end{align} by definition of $\dot x = f(x,u)$. Add the neutral Term to the payoff funtional \begin{align} J(u)&=\int^\infty_0 e^{-\rho t}[F(x,u)+\lambda f(x,u)]dt - \int^\infty_0{e^{-\rho t}\lambda\dot xdt}\\ &=\int^\infty_0 e^{-\rho t}H(x,u,\lambda) - \int^\infty_0{e^{-\rho t}\lambda\dot xdt} \end{align}

Integration by parts of the right term ond the rhs yields \begin{align} \int^\infty_0{e^{-\rho t}\lambda\dot xdt} = [e^{-\rho t}\lambda(t)x(t)]^\infty_0 - \int^\infty_0{e^{-\rho t}x(\dot \lambda-\rho\lambda)dt} \end{align}

Resubstitute that term \begin{align} J(u)=\int^\infty_0 e^{-\rho t}[H(x,u,\lambda) + x(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)x(t) + \lambda(0)x(0) \end{align}

Define \begin{align} x &= x^*+\varepsilon q\\ u &= u^*+\varepsilon p \end{align}

which gives \begin{align} J(\varepsilon)=\int^\infty_0 e^{-\rho t}[H(x^*+\varepsilon q,u^*+\varepsilon p,\lambda) + (x^*+\varepsilon q)(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)[x^*(t)+\varepsilon q(t)] + \lambda(0)x(0) \end{align}

FOC for maximum $J_\varepsilon = 0$ \begin{align} J_\varepsilon=\int^\infty_0 e^{-\rho t}[H_x q + H_u p + q(\dot \lambda-\rho\lambda)]dt - \lim_{t\to\infty}e^{-\rho t}\lambda(t)q(t) = 0 \end{align}

Since $q$ and $p$ are unconstrained we must have \begin{align} H_u &= 0\\ H_x &= \rho\lambda - \dot \lambda\\ \lim_{t\to\infty}e^{-\rho t}\lambda(t) &= 0 \end{align}

Consider the following differential equation \begin{align} \dot x(t)=f(x(t),u(t)) \end{align} where $x$ is the state and $u$ the control variable. The solution is given by \begin{align} x(t)=x_0 + \int^t_0f(x(s),u(s))ds. \end{align} where $x_0:=x(0)$ is the given inital state.

Now consider the following programm \begin{align} &V(x_0) := \max_u \int^\infty_0 e^{-\rho t}F(x(t),u(t))dt\\ s.t.~&\dot x(t)=f(x(t),u(t))\\ &x(0) = x_0 \end{align} where $\rho > 0$ denotes time preference, $V(\cdot)$ is the value and $F(\cdot)$ an objective function. A classical economic application is the Ramsey-Cass-Koopmans model of optimal growth. The Hamilton-Jacobi-Bellman equation is given by \begin{align} \rho V(x)=\max_u [F(x,u) + V'(x)f(x,u)],\quad \forall t\in[0,\infty). \end{align}

Say I've solved the HJB for $V$. The optimal control is then given by \begin{align} u^*=\arg\max_u [F(x,u) + V'(x)f(x,u)]. \end{align} I'll get optimal trajectories for the state and control $\{(x^*(t),u^*(t)):t\in[0,\infty)\}$.

The wiki article says

...but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum.

In Bertsekas (2005) Dynamic Programming and Optimal Control, Vol 1, 3rd ed., in Proposition 3.2.1 he states that solving for $V$ is the optimal cost-to-go function and the associated $u^*$ is optimal. However, he explicitly declares it as a sufficiency theorem.

Actually, I just want to make sure, that if I've solved the HJB and recoverd the associated state and control trajectories, that I don't have to be concerned with any additional optimality conditions.

Addendum: Solution

I think I was able to derive necessary conditions from the maximum principle by the HJB equation itself.

Define the hamiltonian \begin{align} H(x,u,V'(x)) := F(x,u) + V'(x)f(x,u) \end{align}

then we have \begin{align} \rho V(x)=\max_u H(x,u,V'(x)) \end{align}

which is \begin{align} \rho V(x)= H(x,u^*,V'(x)). \end{align}

Define an arbitrary function $q:[0,\infty)\to\mathbb{R}$ with $q(0)=\lim_{t\to\infty} q(t)=0$. Now fix \begin{align} x = x^*+\varepsilon q \end{align}

where $\varepsilon\in\mathbb{R}$ is a parameter. Plug the term into the maximized hamiltonian which gives \begin{align} \rho V(x^*+\varepsilon q)= H(x^*+\varepsilon q,u^*,V'(x^*+\varepsilon q)). \end{align}

At $\varepsilon = 0$ we have the optimal solution. Thus differentite over $\varepsilon$ to get a first order condition \begin{align} \rho V'q = H_x q + H_{V'}V''q. \end{align}

Now define the adjoint variable with \begin{align} \lambda = V'(x). \end{align}

Differentiate over time \begin{align} \dot \lambda = V''\dot x. \end{align}

and note that \begin{align} H_{V'} = f(x,u) = \dot x. \end{align}

Plug everthing into the foc wich gives \begin{align} \rho \lambda = H_x + \dot \lambda. \end{align}

That's it pretty much. So solving the HJB is indeed necessary and sufficient (omitted here) for optimality.

However the transversality condition \begin{align} \lim_{t\to\infty} e^{-\rho t}\lambda(t) = 0 \end{align} is missing.

Consider the following differential equation \begin{align} \dot x(t)=f(x(t),u(t)) \end{align} where $x$ is the state and $u$ the control variable. The solution is given by \begin{align} x(t)=x_0 + \int^t_0f(x(s),u(s))ds. \end{align} where $x_0:=x(0)$ is the given inital state.

Now consider the following programm \begin{align} &V(x_0) := \max_u \int^\infty_0 e^{-\rho t}F(x(t),u(t))dt\\ s.t.~&\dot x(t)=f(x(t),u(t))\\ &x(0) = x_0 \end{align} where $\rho > 0$ denotes time preference, $V(\cdot)$ is the value and $F(\cdot)$ an objective function. A classical economic application is the Ramsey-Cass-Koopmans model of optimal growth. The Hamilton-Jacobi-Bellman equation is given by \begin{align} \rho V(x)=\max_u [F(x,u) + V'(x)f(x,u)],\quad \forall t\in[0,\infty). \end{align}

Say I've solved the HJB for $V$. The optimal control is then given by \begin{align} u^*=\arg\max_u [F(x,u) + V'(x)f(x,u)]. \end{align} I'll get optimal trajectories for the state and control $\{(x^*(t),u^*(t)):t\in[0,\infty)\}$.

The wiki article says

...but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum.

In Bertsekas (2005) Dynamic Programming and Optimal Control, Vol 1, 3rd ed., in Proposition 3.2.1 he states that solving for $V$ is the optimal cost-to-go function and the associated $u^*$ is optimal. However, he explicitly declares it as a sufficiency theorem.

Actually, I just want to make sure, that if I've solved the HJB and recoverd the associated state and control trajectories, that I don't have to be concerned with any additional optimality conditions.

Addendum: Solution

I think I was able to derive necessary conditions from the maximum principle by the HJB equation itself.

Define the hamiltonian \begin{align} H(x,u,V'(x)) := F(x,u) + V'(x)f(x,u) \end{align}

then we have \begin{align} \rho V(x)=\max_u H(x,u,V'(x)) \end{align}

which is \begin{align} \rho V(x)= H(x,u^*,V'(x)). \end{align}

Define an arbitrary function $q:[0,\infty)\to\mathbb{R}$ with $q(0)=\lim_{t\to\infty} q(t)=0$. Now fix \begin{align} x = x^*+\varepsilon q \end{align}

where $\varepsilon\in\mathbb{R}$ is a parameter. Plug the term into the maximized hamiltonian which gives \begin{align} \rho V(x^*+\varepsilon q)= H(x^*+\varepsilon q,u^*,V'(x^*+\varepsilon q)). \end{align}

At $\varepsilon = 0$ we have the optimal solution. Thus differentite over $\varepsilon$ to get a first order condition \begin{align} \rho V'q = H_x q + H_{V'}V''q. \end{align}

Now define the adjoint variable with \begin{align} \lambda = V'(x). \end{align}

Differentiate over time \begin{align} \dot \lambda = V''\dot x. \end{align}

and note that \begin{align} H_{V'} = f(x,u) = \dot x. \end{align}

Plug everthing into the foc wich gives \begin{align} \rho \lambda = H_x + \dot \lambda. \end{align}

That's it pretty much. So solving the HJB is indeed necessary and sufficient (omitted here) for optimality. Someone should add it to wiki. Might save time for people thinking about such problems (won't be a lot I reckon).

However the transversality condition \begin{align} \lim_{t\to\infty} e^{-\rho t}\lambda(t) = 0 \end{align} is missing.

Consider the following Hamilton-Jacobi-Bellman equation \begin{align} \rho V(x)=\max_u [F(x,u) + V'(x)\dot x] \end{align} where $\rho$ denotes time preference, $x$ the state, $u$ the control and $V(.)$ the value Function associated with \begin{align} V(x_0)=\max_u \left[\int^\infty_0 e^{-\rho t}F(x,u)dt\right] \end{align} Say I've solved the HJB for $V(.)$ and I recoverded $u^*$. The wiki article says

...but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum.

However, in Bertsekas (2005) Dynamic Programming and Optimal Control, Vol 1, 3rd ed., in Proposition 3.2.1 he states that solving for $V$ is the optimal cost-to-go function and the associated $u^*$ is optimal. However, he explicitly declares it as a sufficiency theorem.

Actually, I just want to make sure, that if I've solved the HJB and recoverd the associated state and control trajectories, that I don't have to be concerned with any additional optimality conditions.

Consider the following differential equation \begin{align} \dot x(t)=f(x(t),u(t)) \end{align} where   $x$ is the state and $u$ the control variable. The solution is given by \begin{align} x(t)=x_0 + \int^t_0f(x(s),u(s))ds. \end{align} where $x_0:=x(0)$ is the given inital state.

Now consider the following programm \begin{align} &V(x_0) := \max_u \int^\infty_0 e^{-\rho t}F(x(t),u(t))dt\\ s.t.~&\dot x(t)=f(x(t),u(t))\\ &x(0) = x_0 \end{align} where $\rho > 0$ denotes time preference, $V(\cdot)$ is the value and $F(\cdot)$ an objective function. A classical economic application is the Ramsey-Cass-Koopmans model of optimal growth. The Hamilton-Jacobi-Bellman equation is given by \begin{align} \rho V(x)=\max_u [F(x,u) + V'(x)f(x,u)],\quad \forall t\in[0,\infty). \end{align}

Say I've solved the HJB for $V$. The optimal control is then given by \begin{align} u^*=\arg\max_u [F(x,u) + V'(x)f(x,u)]. \end{align} I'll get optimal trajectories for the state and control $\{(x^*(t),u^*(t)):t\in[0,\infty)\}$. 

The wiki article says

...but when solved over the whole of state space, the HJB equation is a necessary and sufficient condition for an optimum.

In Bertsekas (2005) Dynamic Programming and Optimal Control, Vol 1, 3rd ed., in Proposition 3.2.1 he states that solving for $V$ is the optimal cost-to-go function and the associated $u^*$ is optimal. However, he explicitly declares it as a sufficiency theorem.

Actually, I just want to make sure, that if I've solved the HJB and recoverd the associated state and control trajectories, that I don't have to be concerned with any additional optimality conditions.

Addendum: Solution

I think I was able to derive necessary conditions from the maximum principle by the HJB equation itself.

Define the hamiltonian \begin{align} H(x,u,V'(x)) := F(x,u) + V'(x)f(x,u) \end{align}

then we have \begin{align} \rho V(x)=\max_u H(x,u,V'(x)) \end{align}

which is \begin{align} \rho V(x)= H(x,u^*,V'(x)). \end{align}

Define an arbitrary function $q:[0,\infty)\to\mathbb{R}$ with $q(0)=\lim_{t\to\infty} q(t)=0$. Now fix \begin{align} x = x^*+\varepsilon q \end{align}

where $\varepsilon\in\mathbb{R}$ is a parameter. Plug the term into the maximized hamiltonian which gives \begin{align} \rho V(x^*+\varepsilon q)= H(x^*+\varepsilon q,u^*,V'(x^*+\varepsilon q)). \end{align}

At $\varepsilon = 0$ we have the optimal solution. Thus differentite over $\varepsilon$ to get a first order condition \begin{align} \rho V'q = H_x q + H_{V'}V''q. \end{align}

Now define the adjoint variable with \begin{align} \lambda = V'(x). \end{align}

Differentiate over time \begin{align} \dot \lambda = V''\dot x. \end{align}

and note that \begin{align} H_{V'} = f(x,u) = \dot x. \end{align}

Plug everthing into the foc wich gives \begin{align} \rho \lambda = H_x + \dot \lambda. \end{align}

That's it pretty much. So solving the HJB is indeed necessary and sufficient (omitted here) for optimality.

However the transversality condition \begin{align} \lim_{t\to\infty} e^{-\rho t}\lambda(t) = 0 \end{align} is missing.