# When Optimal Control fails (?)

In order to "ask my question", I have to solve a model first. I will omit some steps but still, this will unavoidably make this post very long -so this is also a test to see whether this community likes such kind of questions.

Before starting, I clarify that this may look totally like a standard neoclassical growth model in continuous time, but it is not: It is concerned with a single individual, which does not "represent" anybody else in the economy around him, an economy which is not modeled. The framework here is "application of Optimal Control to the maximization problem of a single individual". This is about the Optimal Control solution framework and method itself.

We solve the intertemporal utility maximization problem of a small businessman that owns the capital in his firm, while he purchases labor services in a perfectly competitive labor market, and he sells his product (fresh doughnuts) in a perfectly competitive goods market. We set the model in continuous time without uncertainty (socioeconomic conditions are steady), and with infinite horizon (the businessman envisions many future copies of him in a row):

$$\max_{c,\ell,k}\int_0^{\infty}e^{-\rho t}\ln c\,\text{d}t\\ \text{s.t.}\;\; \dot k = f(k,\ell) - w\ell - \delta k - c\\ \lim_{t\rightarrow \infty}e^{-\rho t}\lambda(t) k(t) = 0$$

where $c$ is the businessman's consumption, $\ln c$ is instantaneous utility from consumption, $\rho>0$ is the rate of pure time preference, $k$ is the firm's capital, $\delta$ is the capital depreciation rate, and $f(k,\ell)$ is the production function of the business. Initial level of capital is given, $k_0$. The businessman's own occupation with the business is subsumed into capital. The production function is standard neoclassical (constant returns to scale, positive marginal products, negative second partials, Inada conditions). The constraints are the law of motion of capital, and the Transversality condition using the current value multiplier.

Setting up the current value Hamiltonian

$$\hat H = \ln c +\lambda[f(k,\ell) - w\ell - \delta k - c]$$

we calculate the first-order conditions

$$\frac {\partial \hat H}{\partial c} = 0 \Rightarrow \frac 1c =\lambda \Rightarrow \frac {\dot c}{c} = -\frac {\dot \lambda}{\lambda}$$

$$\frac {\partial \hat H}{\partial \ell} = 0 \Rightarrow \lambda [f_{\ell} -w]=0 \Rightarrow f_{\ell} =w$$

$$\frac {\partial \hat H}{\partial k} = \rho\lambda - \dot \lambda \Rightarrow \lambda [f_{k} -\delta]=\rho\lambda - \dot \lambda$$

and combining them we obtain the law of evolution of consumption of our businessman ,

$$\dot c = \big(f_k-\delta -\rho \big)c \tag{1}$$

From the optimal rule for labor demand $\ell: f_{\ell} =w$ (static) and the constant returns to scale implication ($f = f_kk + f_{\ell}\ell$) we obtain $f-w\ell = f_kk$. Inserting this into the law of motion of capital we obtain

$$\dot k = f_kk - \delta k - c \tag {2}$$

Equations $(1)$ and $(2)$ form a system of differential equations. The steady-state values for the consumption and the capital of the businessman are

$$c^* = f_k^*k^* - \delta k^*,\;\;\; k^*: f^*_k = \delta +\rho \tag{3}$$

$$\Rightarrow c^* = \rho k^* \tag{3a}$$

...which is a pretty familiar expression.

$k^*$ is sometimes called the "modified golden rule" level of capital. The Jacobian of the system evaluated at the steady state values has a negative determinant for any value of the model parameters, which is a necessary and sufficient condition for the system to exhibit saddle-path stability.

The maximum of the $\dot k =0$ locus is at the point, $\tilde k$ (sometimes called the "golden rule" level of capital)

$$\tilde k : f_{kk}(\tilde k)\tilde k + f_k(\tilde k) - \delta = 0\Rightarrow f_k(\tilde k) = \delta - f_{kk}(\tilde k)\tilde k \tag{4}$$

The $\tilde k$ value is important as a benchmark: it is the level of capital where $\dot k =0$ and $c$ is at a maximum (not optimal or steady state).

The $\dot c=0$ loci crosses the horizontal axis of the phase diagram (that measures capital) at the steady-state capital level $k^*$.

If $k^* > \tilde k$, which requires $f_k^* < f_k(\tilde k)$ due to negative second partials, we will have "over-accumulation of capital" (too many doughnuts): the businessman could enjoy more steady-state consumption with a lower level of capital. Using $(3)$ and $(4)$ we have

$$f_k^* < f_k(\tilde k) \Rightarrow \delta + \rho < \delta - f_{kk}(\tilde k)\tilde k$$

$$\Rightarrow \rho < - f_{kk}(\tilde k)\tilde k \tag {5}$$

Inequality $(5)$ is the condition for sub-optimal steady-state level of capital. And the thing is, we cannot rule it out. It simply requires that the businessman is "sufficiently patient", with a sufficiently small rate of pure time preference, but still positive.

Here starts the problem: overaccumulation of capital is effectively excluded in the representative agent model. It is possible in overlapping generation models, but as an unintended consequence at the macroeconomic level, one of the earliest examples that the macro-economy may be micro-founded and still behave differently than the micro-world.

But our model falls in neither category: it is a partial equilibrium model of a single agent in an implicitly heterogeneous environment -and general equilibrium here won't alter the results: this person represents only himself. So the problem is that if $(5)$ holds, then the Optimal Control solution will be obviously sub-optimal, because here we have a single person, a single will, a single mind: by looking at the solution our businessman will say, "hey, this method is worthless, if I follow its advice I will end up with a sub-optimally high level of capital".

And I am not satisfied to simply say "well, Optimal Control is not suitable for this problem, try another method", because I cannot see why we should consider it unsuitable. But if it is suitable, then the method should signal that something is wrong, it should at some point require that $(5)$ does not hold, in order to be able to offer a solution (if it so happens that $(5)$ does not hold, everything looks swell).

One could wonder "maybe the Transversality condition is violated if $(5)$ holds?" -but it doesn't look that it does, since $\lambda(t)k(t) = k(t)/c(t)$, which goes to a positive constant, while $e^{-\rho t}$ goes to zero, requiring only that $\rho>0$.

My questions:

1) Can somebody offer some insight here?

2) I would be grateful if somebody solved this using Dynamic Programming and reported the results.

From a mathematical point of view, the crucial difference of this model is that the optimized law of motion of capital, eq. $(2)$ includes not the whole output $f(k)$ as in the standard model, but only the returns to capital $f_kk$. And this happens because we have separated property rights over the output, which in the "individual business maximization problem" framework, is to be expected.

• I'm not sure what you mean when you say "the maximum of the kdot=0 locus". Maximum with respect to what? Also, when you calculate (4), shouldn't you be totally differentiating (2)--i.e. shouldn't you also calculate the change in c that is necessary to ensure kdot=0 is still satisfied after you change k? – Ubiquitous Nov 21 '14 at 15:21
• @Ubiquitious Maximum with respect to capital. This is exactly how phase diagrams are drawn, but I could not include also these calculations here. For the second question: $(4)$ comes from setting $\dot k =0$ in $(2)$ and expressing consumption as a function of capital, $c = f_kk-\delta k$ (not evaluated at the steady state value). To obtain the shape of this locus, we differentiate it with respect to capital. – Alecos Papadopoulos Nov 21 '14 at 15:27
• I haven't checked the whole thing, but one problem I see is that labor optimality condition will (under CRS) determine capital/labor ratio, which in turn determines marginal product of capital, which will thus be constant along the optimal path. The model is then equivalent to standard consumption-saving problem with exogenous interest rate, so if MPK - delta > rho, the agent's consumption will be growing at constant rate (i.e. there's no steady state). – ivansml Dec 18 '14 at 18:12
• @ivansml. Thanks for contributing. But the solution does not say that $f_k -\delta >\rho$. The steady state is at the point where $f_k -\delta = \rho$, eq. $(3)$. The problem is to what level of capital this steady-state corresponds, and whether it will be above or below the "golden rule" level $\tilde k$. – Alecos Papadopoulos Dec 18 '14 at 18:18
• Only now I noticed this question is rather old... hope that doesn't matter. Back to topic - $f_k$ must be determined by labor FOC. Steady state will exist only if this value of $f_k$ also equals $\rho+\delta$, i.e. by coincidence (or some general equilibrium consideration). If it is higher, agent will accumulate capital indefinitely and his consumption grows, if it is lower, he will decumulate capital and his consumption falls. It's really all due to CRS assumption - the "revenue" function $f(k,\ell) - w \ell$ is linear in $k$ once the firm optimizes over labor, so steady growth is possible. – ivansml Dec 18 '14 at 18:35

I believe the problem is that the steady state may not exist, and the system instead exhibits steady growth (depending on parameters).

The reason is because the model is equivalent to the standard consumption-saving problem with exogenous and constant interest rate. To see that, first consider the first order condition for labor choice $f_2(k,\ell) = w$ (here, $f_i$ is partial derivative of $f$ wrt. $i$th argument). Using the definition of constant returns, marginal product of labor is $$\frac{\partial }{\partial \ell} f(k,\ell) = \frac{\partial }{\partial \ell} \left[ f \left( \frac{k}{\ell},1 \right) \ell \right] = f_1 \left( \frac{k}{\ell},1 \right) \frac{-k}{\ell} + f \left( \frac{k}{\ell},1 \right)$$ which is a function of capital-labor ratio only. If wage is constant, labor FOC determines uniquely the optimal $k/\ell$ ratio as a function of wage $w$ and other parameters. Since marginal product of capital $$\frac{\partial }{\partial k} f(k,\ell) = \frac{\partial }{\partial k} \left[ f \left( \frac{k}{\ell},1 \right) \ell \right] = f_1 \left( \frac{k}{\ell},1 \right)$$ also depends on $k/\ell$, it will be constant along optimal path. Denote this value of marginal product $r^*$, and denote the return net of depreciation $r = r^* - \delta$. Equations (1) - (2) for dynamics of capital and consumption are then $$\begin{split} \dot c_t &= (r - \rho) c_t \\ \dot k_t &= r k_t - c_t \end{split}$$ and the specific solution which satisfies transversality condition should be $c_t = \rho k_t$ with $k_0$ given, i.e. a constant part of wealth is consumed at each moment. Both capital and consumption grow at rate $(r-\rho)$, so there is no steady state unless the return on capital (which here depends on exogenous wage rate $w$) equals rate of time preference.

• (+1) Thank you. I am taking this up now into an answer of mine. – Alecos Papadopoulos Dec 18 '14 at 23:51
• great answer. basically, once labor is chosen optimally, the profit function becomes linear in capital - so that this model boils down to an AK model, whose properties (including steady state growth) are well understood. – nominally rigid Dec 19 '14 at 6:42
• @nominallyrigid But only if we assume that the wage stays constant. Remember this is not general equilibrium, just a tiny individual swimming in the economy's ocean. – Alecos Papadopoulos Dec 19 '14 at 19:52

I am posting this as an answer, because it continues on user @ivansml answer... which is the one that identified the catch here, a catch I naively have overlooked (although it is a narrow case, while the interesting par comes after. Nevertheless, it should have been dealt with).

Indeed, with exogenous wage rate, and perfectly competitive optimization on labor demand, the marginal product of capital is determined only by the parameters of the model and by the wage rate. For the simple case where we assume that the wage rate is constant the analysis of @ivansml holds: the model becomes one of endogenous growth: the marginal product of capital is constant, which is what is needed for endogenous growth, where there is no steady state in levels.

Denoting $\hat c = \dot c/c$ and $\hat k = \dot k /k$ , equations $(1)$ and $(2)$ of the OP can be written

$$\hat c = f_k-\delta -\rho \tag{1b}$$ $$\hat k = f_k - \delta - c/k \tag{2b}$$

Since $f_k$ is constant, the growth rate of consumption is constant - zero, positive, or negative, depending on the parameters and the wage. On the other hand differentiating $(2b)$ with respect to time we get

$$\dot {\hat k} = (\hat k - \hat c)\cdot (c/k)$$

and it is obvious that for steady-state growth we want $\hat k = \hat c$, which, from $(2b)$ is obtained only if $c = \rho k$. It is easy to verify that, since $\lambda(t) = c(t)$, the only way that the transversality condition will hold, is if consumption and capital grow, or shrink, at the same rate (or stay constant).

In endogenous growth models proper where we examine the whole economy, we just assume that the parameters of the model are such that there is a positive growth rate, because this is what we observe in the real world. But here, we have just one individual. So, what we may be telling our businessman?

If $f_k-\delta -\rho >0$, the growth rate is positive, and both his consumption and capital should grow "forever", maintaining a constant ratio.
If $f_k-\delta -\rho =0$, the growth rate is zero, and both variable stay forever constant.
If $f_k-\delta -\rho <0$, the growth rate is negative, and we should enter into a downward spiral of diminishing consumption and capital (always maintaining the relation $c= \rho k$).

This, has some intuition, validating the appropriateness of Optimal Control application: given the other parameters and the wage rate, the larger the "impatience" (the larger $\rho$ is) the more possible becomes that the individual will experience diminishing consumption levels, since the future, and hence investment, are not much to his liking. Of course, a monotonic downward spiral may not sound very realistic as a solution -but this is a very stylized model, providing essentially general tendencies in a necessarily highly formal mathematical language.

The really interesting part will start if we consider a variable wage. This can create all sorts of interesting and complicated dynamics for our little businessman and his consumption-investment decisions.

I think that the key question is whether this firm is the only firm in the economy. If it is then it is no longer correct for it to take $w$ as given as $w$ will be affected by its own capital accumulation decision. In this case you should make the substitutions that you made before your equation (2) while setting up the Hamiltonian. On the other hand if this is one of many firms, so that the wage rate is exogenous, then the substitutions before eq. (2) are not valid. I think you need to carefully distinguish between big-$k$, the aggregate capital in the economy, and little-$k$ the capital chosen by this decision maker.

• I am strictly looking at a single firm that remains too small to influence the aggregate. So your second comment is relevant, where you say "the substitutions before equations (2) are not valid". I don't see why. Can you elaborate (preferably formally) on that please? Thank you. – Alecos Papadopoulos Nov 21 '14 at 14:44
• @AlecosPapadopoulos I think the problem is not mathematical but one of interpretation. If my firm is too small to influence the economy, why should it be the case that $w=f_l$ or $r=f_k$ for my firm regardless of the $k$ I choose, which seems to be the assumption implicit in the substitutions you make before (2) and then differentiating the RHS of the $\dot k$ equation with respect to $k$. – Jyotirmoy Bhattacharya Nov 21 '14 at 15:18
• @JyotirmoyBhattacharya that's a standard result from assuming competitive markets. – FooBar Nov 21 '14 at 15:19
• @FooBar In a competitive market you choose $k$ and $l$ to make $w=f_l$ and $r=f_k$. The conditions do not hold at arbitrary $l$ and $k$. – Jyotirmoy Bhattacharya Nov 21 '14 at 15:21
• Ok, I will have to write the Hamiltonian after all, and make this even lengthier. – Alecos Papadopoulos Nov 21 '14 at 15:31