Remainder term in Linear Approximations going to 0

Question

A number of proofs in optimisation use the idea that the remainder term in either the differential or the Taylor Approximation go to zero. For example:

1. Some envelope theorem proofs:.
2. Necessity and Sufficiency of FOC & SOC in optimisation proofs.

I'm struggling to understand the Remainder term going to 0 as explained below.

1) Removing the remainder term from the differential

Used in an envelope theorem proof:

Lets say we have a constraint function $G(\mathbf{(x(θ), θ)} =b$

The differential for $G$ when $θ _ι$  changes is:

$∆G = \sum_{j=1}^{n}[\frac{\partial{G}}{\partial{x_j}}∆x_j] + \frac{\partial{G}}{\partial{θ_i}}∆θ_i +R = 0$ Where R is our remainder term:

Then dividing through by $∆θ_i$ and taking the $\lim \limits_{∆θ_i \to 0}$ we get:

$\lim \limits_{∆θ_i \to 0} \frac{∆G}{∆θ_i} = \lim \limits_{∆θ_i \to 0} \sum_{j=1}^{n}[\frac{\partial{G}}{\partial{x_j}}\frac{∆x_j}{∆θ_i}] + \frac{\partial{G}}{\partial{θ_i}} +\frac{R}{∆θ_i} = 0$

• Question: How do I show that: $\lim \limits_{∆θ_i \to 0} \frac{R}{∆θ_i} = 0$

I understand why R would go to 0. Because as the change approaches 0 then $f(x) = P(x)$ Where $P(x)$ is our Taylor Polynomial, and hence the difference $R(x)$ goes to 0.

But then don't we have $\frac{0}{0}$ and i don't know how to resolve this in this case?

2) Removing the remainder term from the second order Taylor Polynomial

I've seen this written as Taylors Theorem I've seen this be used to prove the necessity and sufficiency of FOC and SOC.

If we define $R_2(∆x) = f(x) - P_2(x)$ where $P_2(x)$ is the second order Taylor polynomial around $a$.

• Question: How do we show that: $\lim \limits_{∆x_i \to 0} \frac{R_2(∆x)}{(∆x)^2} = 0$

Defining $∆x = x - a$ My best effort so far from expanding $f(x)$ and $P_2(x)$, we get:

$\lim \limits_{∆x_i \to 0} [\frac{f'(a)}{∆x} - \frac{f'(a)}{∆x} - \frac{1}{2}f''(a)]$ = $- \frac{1}{2}f''(a)$ = $- \frac{1}{2}f''(x) ≠ 0$

If anyone can help with these it would solve many riddles for me. Expanding to $\mathbb{R^n}$ and other generalised cases I should then be able to do myself, thanks!

Answer 1

• Question: How do I show that: $\lim \limits_{∆θ_i \to 0} \frac{R}{∆θ_i} = 0$

[...]

• Question: How do we show that: $\lim \limits_{∆x_i \to 0} \frac{R_2(∆x)}{(∆x)^2} = 0$

As you can see from your questions, it must be proved not only that the remainders $R_1$ or $R_2$ go to zero, but the ratio between the remainders and $(\Delta x) ^2$ or $\Delta \theta$ goes to zero.

That is, the remainder $R_n$ in Taylor's formula is an infinitesimal of greater order with respect to $(\Delta x)^n$. This is the important theorem that must be proved with regard to Taylor's formula.

Intuitively, this means that the remainder $R_n$, of a Taylor's polynomial of order $n$,$^1$ goes to zero more 'quickly' tha $(\Delta x)^n$.

Therefore, it is not sufficient to prove that $R_n$ goes to zero, because it can go to zero 'slowly', so that the ratio $\frac {R_n}{\Delta x} \nrightarrow 0$.

This is important, as a Taylor's polynomial is an approximation of the function at a point: for example, intuitively, if when $\Delta x$ is very small the remainder is very high, we haven't a good approximation.

As you rightly noticed, this ratio, as $\Delta x \rightarrow 0$, is the indeterminate form $0/0$. Therefore, the proof uses L'Hôpital's rule.

I give you below a proof that assumes that the derivatives of the function are continuos, but a similar proof exists for the general case.$^2$

$$***$$

THEOREM: TAYLOR'S FORMULA. - Let $f: \mathbb{R} \rightarrow \mathbb{R}$ be a function and the derivatives $f^{(n)}$ exist and are continuous at a point $x_0$.

We have

$$f(x)= \sum_{k=0}^{n}\frac { f^{(k)}(x_0)} {k!}(x-x_o)^k+R_n(x) \qquad (1)$$

and

$$\lim _{x\rightarrow x_0} \frac {R_n(x)}{(x- x_0)^n}=0 \qquad (2).$$

PROOF.

According to the definition of $R_n$ we must prove that:

$$\lim _{x\rightarrow x_0} \frac {f(x) -[f(x_0) + f'(x_0) (x-x_0)+ ...+f^{(n)} (x_0)(x-x_0)^n/n!]}{(x- x_0)^n}=0 \qquad (3)$$

We now use l'Hôpital rule. The derivative is respect to $x$, so the derivative of $f(x_0)$ is zero, whereas the derivative of $\frac {f^{(n)}(x-x_0) ^n}{n!}$ is

$$ \frac {f^{(n)}(x_0)n (x-x_0) ^{n-1}}{n!}= \frac {f^{(n)}(x_0) (x-x_0) ^{n-1}}{(n-1)!}\qquad (4)$$

Therefore, the limit $(3)$, having applied l'Hôpital, is the same as

$$\lim _{x\rightarrow x_0} \frac {f'(x) -[f'(x_0) + ...+f^{(n)} (x_0)(x-x_0)^{n-1}/(n-1)!]}{n(x- x_0)^{n-1}}. \qquad (5)$$

If $n>1$, we have again an indeterminate form $0/0$. After applying $n$ times l'Hôpital rule, we have:

$$\lim _{x\rightarrow x_0} \frac {f^{(n)} (x)-f^{(n)} (x_0)}{n!} \qquad (6).$$

This last limit is zero, as $f^{(n)} (x)$ is continuous, by assumption, at $x_0$.

$\Box$

If reading the proof with a generic $n$ is too cumbersome, you can re-read the method of the proof with $n=2$.

---

$^1$ ^{This result holds for every order $n$ of the Taylor's polynomial.}

$^2$ ^{You can find this proof and the overall theory of the Taylor's polynomial in a good book of calculus/ undergraduate text of mathematical analysis.}