# How can I apply the Hamiltonian function and Pontryagin's maximum principle in the context of Optimal Control Theory?

I am really struggling to grasp how the Hamiltonian Function and Pontryagin's Maximum Principle work in the context of Optimal Control Theory (Maths for Economics) course. I am given the following conditions:

$$\quad\space\max_{u} \mathcal{F} = \int_{t_0}^{t_f} f(x(t), u(t), t) \, dt + S(x_f) \tag{1.1}$$

$$\begin{cases} \begin{array}{l} \dot x(t) = g(x(t),u(t),t) \\ x(t_i)=x_i \\ x(t_f)=x_f \\ \end{array} \end{cases} \qquad \begin{array}{l} \text(constraints) \\ \text(known) \\ \text(unknown) \\ \end{array} \tag{1.2}$$

$$\begin{cases} \begin{array}{l} \overline{x} = (x_1,x_2,\dots,x_n) \\ \overline{u} = (u_1,u_2,\dots,u_n) \\ \overline{g} = (g_1,g_2,\dots,g_n) \\ \end{array} \end{cases} \qquad\quad \begin{array}{l} \text(state \space variables) \\ \text(control \space variables) \\ \text(constraint \space functions) \\ \end{array} \tag{1.3}$$

So far I understand that we construct the Hamiltonian $$\mathcal{H}$$ from the Hamiltonian Functional $$\mathcal{L}$$ and the costate functions $$\lambda$$ such that:

$$\mathcal{H}(x,u,t,\lambda) = f(x,u,t) + {\lambda}(g(x,u,t)) \tag{2.1}$$

Then, we find the Hamiltonian equations for the state $$x$$ and costate $$\lambda$$ variables:

$$\begin{cases} \dot{x} = \frac{\partial \mathcal{H}}{\partial \lambda} \\ \dot{x}_i = \frac{\partial \mathcal{H}}{\partial x_i} \quad \text{for} \quad n > 1 \end{cases} \tag{3.1}$$

$$\begin{cases} \dot{\lambda} = -\frac{\partial \mathcal{H}}{\partial x} \\ \dot{\lambda}_i = -\frac{\partial \mathcal{H}}{\partial x_i} \quad \text{for} \quad n > 1 \\ \lambda_f = \frac{\partial \mathcal{H}}{\partial x_f} S(x_f) \end{cases} \tag{3.2}$$

We are then instructed to solve using Pontryagin's Maximum Principle:

$$\text{If } m = 1 \begin{cases} \frac{\partial \mathcal{H}}{\partial u} = 0\\ \frac{\partial^2 \mathcal{H}}{\partial u^2} < 0 \\ \end{cases} \tag{4.1}$$

$$\text{If } m > 1 \begin{cases} \nabla \mathcal{H} = \overline{0} \\ \nabla^2 \mathcal{H} < 0 \quad \text{for} \quad \text{odd } |M_H| \\ \nabla^2 \mathcal{H} > 0 \quad \text{for} \quad \text{even } |M_H| \\ \end{cases} \tag{4.2}$$

Finally, we are told that if we only have the initial conditions $$x(t) = x_i$$, then:

$$\begin{cases} x(t_i) = x_i \\ \lambda(t_f) = \frac{\partial \mathcal{H}}{\partial x_f} S(x_f) \quad \text{(transversality condition)} \\ \end{cases} \tag{5.1}$$

While, if we have both initial and boundary conditions, then:

$$\begin{cases} x(t_i) = x_i \\ x(t_f) = x_f \\ \end{cases} \tag{5.2}$$

I fail to understand the following points:

• In the Hamiltonian equations for the state variables 3.1, where did the second equation come from and how did it go from being the derivative of $$\mathcal{H}$$ with respect to $$\lambda$$ to being the derivative of $$\mathcal{H}$$ with respect to $$\mathcal{x_i}$$? Shoulnd't it derivate with respect to $$\lambda$$ as well?

• In the Hamiltonian equations for the costate variables 3.2, where does $$\lambda_f$$ come from? When is it used?

• In equations 4.1 and 4.2, what does $$m$$ stand for? I understand from my professors lectures that it means multiplicity, but in a different sense than when used, for example, to talk about eigenvalues in the context of dynamical systems. What, then, is it alluding to?

• In equation 5.1, what is the transversality condition, and when/how is it applied? Why is it not necessary when boundary conditions are present, as shown in 5.2?

Overall, I am currently unable to solve this kind of problem. Any help understanding how to approach it would be greatly appreciated. As a quick sidenote, any appropiate criticism on the formatting of this post and/or my usage of LaTex would also be appreciated.

• Are you sure there aren’t typos in your formulas? In $(3.1)$, the second equation should be indeed $\dot{x}_i = \frac{\partial \mathcal{H}}{\partial \lambda_i},\; i=1,...,n$. In $(3.1)$ the first equation relates to the case there is only one constraint $g$, that is $n=1$, the second equation to the case we have $n>1$ constraints $g_i$. Jan 24 at 12:36
• As for transversality conditions, they are present in the so-called moving-end problems, that is when the value of your $x(t)$ at the final time $t_f$ is not fixed, and they are not present in the fixed-end problems, when you know the final value of $x(t)$ as in $(5.2)$. Jan 24 at 12:38
• @ astute-hoplite Transversality conditions arise in moving ends problems (and also in infinite time problems) and they are mathematical conditions for a maximum, necessary conditions in the case of moving ends. Their explanation is not trivial, as all proofs of the maximum principle. They have a geometrical meaning, they arise when the arrival of the trajectory is a so called manifold, think of it as a surface in $\mathbb{R}^n$, and not a point. In a sense, from a pratical point of view, yes, they are used as one uses a boundary conditions, but they are actual optimum conditions. Jan 25 at 7:51
• The equation $\lambda_f = \frac{\partial \mathcal{H}}{\partial x_f} S(x_f)$ in $(3.2)$ arises in the so-called Bolza Problem, where the functional to be minimized is the sum of an integral and a function depending on the boundary, in your case the final value $x_f$. It is a problem of Calculus of Variation, that has been dealt with the Pontryagin’s Maximum Principle, see encyclopediaofmath.org/wiki/Bolza_problem Jan 25 at 13:50
• I synthetise my comments in an answer, for clarity, and because this way your question has an answer. Jan 25 at 15:21

I give below just an outline to orient oneself in the context of Pontryagin’s Principle, answering briefly your questions.

First of all, it must be noticed that what is called the Pontryagin's Maximum Principle is actually not a single theorem, but a collection of theorems, that differ according to different assumptions.

In particular, various cases of the Maximum Principle exist according to the interval of time under consideration, which can be finite or infinite (the second extreme of integration becoming $$\infty$$), and the characteristics of the boundary conditions we have.

In your case you have a finite time problem, and as for boundary conditions a distinction must be made between fixed end problems and moving end problems. $$\;$$

In the Hamiltonian equations for the state variables 3.1, where did the second equation come from?

Are you sure there aren’t typos in your formulas? In $$(3.1)$$, the second equation should be indeed $$\dot{x}_i = \frac{\partial \mathcal{H}}{\partial \lambda_i}\; i=1,...,n$$.

In $$(3.1)$$ the first equation relates to the case there is only one constraint $$g$$, that is $$n=1$$, the second equation to the case we have $$n>1$$ constraints $$g_i$$. $$\;$$

In equation 5.1, what is the transversality condition, and when/how is it applied? Why is it not necessary when boundary conditions are present, as shown in 5.2?

Transversality conditions arise only in moving ends problems (and also in infinite time problems) and are absent in fixed end problems: they are present in the so-called moving-end problems, that is when the value of your $$x(t)$$ at the final time $$t_f$$ is not fixed, and they are not present in the fixed-end problems, when you know the final value of $$x(t)$$, as in $$(5.2)$$.

Transversality conditions are mathematical conditions for a maximum, necessary conditions in the case of moving ends.

Their explanation is not trivial, as all proofs of the maximum principle. They have a geometrical meaning, they arise when the arrival of the trajectory is a so called manifold, think of it as a surface in $$\mathbb{R}^n$$, and not a point.

In a sense, yes, from a practical point of view, they are used as one uses boundary conditions, but they are actual optimum conditions.

Proofs of the Maximum Principle are complex, consider that the proof of the main theorem in Pontryagin's original book$$^1$$ takes forty pages, the whole second chapter. $$\;$$

In the Hamiltonian equations for the costate variables 3.2, where does $$\lambda_f$$ come from? When is it used?

The equation $$\lambda_f = \frac{\partial \mathcal{H}}{\partial x_f} S(x_f)$$ in $$(3.2)$$ arises in the so-called Bolza Problem, where the functional to be minimized is the sum of an integral and a function depending on the boundary, in your case (eq.$$1.1$$) a function $$S(x_f)$$ of the final value $$x_f$$ (remember that in this problem the final value $$x_f$$ is a variable, it is not fixed).

It is a problem of Calculus of Variation, that has been dealt with the Pontryagin’s Maximum Principle, see
https://encyclopediaofmath.org/wiki/Bolza_problem

$$\;$$

In equations 4.1 and 4.2, what does $$m$$ stand for? I understand from my professors lectures that it means multiplicity

As for $$m$$ in $$(4.1)$$ and $$(4.2$$) are you sure that it is not $$n$$ as in $$(3.1)$$ ? I see a gradient for $$m>1$$ and a partial derivative for $$m=1$$.

$$^1$$ Pontryagin L.S. and others, The Mathematical Theory of Optimal Processes, Pergamon Press, 1964.

• thank you! Your help has been crucial in terms of understanding this part of my course's curriculum. Jan 25 at 17:41
• I'm happy having ben useful! Pontryagin maximum principle is not an easy subject. Jan 25 at 17:43
• @BakerStreet :Would you mind mentioning ( should I start a new thread ? ) what texts you used that helped lead to your knowledge. Your answer above is MUCH, MUCH appreciated. I've never seen such an eloquent "seeing the forest through the trees" explanation of this material. Dorfman also wrote a nice, intro paper on the hamiltonian from an economic point of view. If I can find a free version, I'll send a link. Jan 26 at 4:15
• There was a free version of this somewhere at some point but now I can only find it on JSTOR. jstor.org/stable/1810679 Note that I mis-stated the title. It's "optimal control theory from an economic point of view" rather than the Hamiltonian. But there is material regarding the Hamiltonian in the paper. Jan 26 at 4:23
• Thanks BakerStreet. I have Liberzon in storage so I will someday read that. I don't have Takayama so I will check it out. Yes, Pontryagin's text is DIFFICULT. Thank you again for your great answer and I hope you like Dorfman's paper. It's basic but I found it to be a nice intro and way of looking at things. Jan 26 at 13:43