6
$\begingroup$

Many presentations of OLS have a condition, $E(X'X)<\infty$ and is invertible. My question is, why is $< \infty$ critical?

Consider,

$$y_i =\beta_0 +\beta_1 x_{1i} +\varepsilon_i $$

The condition, $E(X'X)<\infty$ in this setting is equivalent to $$Var(x_1)<\infty$$ and $E[x_1]<\infty$.

The OLS estimator of $\beta_1$ is $$\hat{\beta_1}=\frac{Cov(x,y)}{Var(x)} = \frac{Cov(x,\beta_0 +\beta_1 x_{1i} +\varepsilon_i)}{Var(x)} =\beta_1 \frac{Var(x)}{Var(x)} + \frac{Cov(x,u)}{Var(x)}$$

Even if $Var(x)=\infty$, we have $Var(x)/Var(x)=1 $ $$\hat{\beta_1}=\beta_1 + \frac{Cov(x,u)}{Var(x)}$$

The way I see it, if $Cov(x,u)=0$, then $0/\infty$ is typically defined as 0 and there are no issues.

Thanks in advance!

$\endgroup$
5
  • $\begingroup$ This would be squarely on topic at Cross Validated and a duplicate question might have already been answered there. $\endgroup$ Dec 20, 2021 at 15:14
  • $\begingroup$ As far as I know $\infty/\infty$ is not well defined and not necessarily equal to 1: e.g. $\lim_{x \to \infty} \frac{x^2}{x}$. $\endgroup$
    – tdm
    Dec 20, 2021 at 15:52
  • $\begingroup$ @tdm sure, but at what point does $\infty/\infty$ become relevant in this problem? $0/\infty$ is typically defined as 0 right? $\endgroup$ Dec 20, 2021 at 16:27
  • $\begingroup$ I hope we can agree $lim_{x\rightarrow \infty} \frac{x}{x} =1$ $\endgroup$ Dec 20, 2021 at 16:39
  • $\begingroup$ Perhaps stats.stackexchange.com/questions/243452/… can help. $\endgroup$ Dec 20, 2021 at 23:35

2 Answers 2

4
$\begingroup$

So let us begin with a real case where the variance has to be infinite, the stock market.

I am not doing a complete derivation here as it would run as multiple chapters. I know this because I am doing web presentations on this. But I can do enough to show you that there cannot be a first moment for the stock market. From that, I will explain the consequence for regressors.

Let us begin with a couple of simple observations about a return or a reward for investing.

If we define a return on a single pair of cash flows as $$r_t=\frac{p_{t+1}}{p_t}\times\frac{q_{t+1}}{q_t}-1$$ and $$R_t=r_t+1$$ we can note a few things.

First, $$R_t=R(p_t,p_{t+1},q_t,q_{t+1}).$$ By definition, prices are data. Volumes are data. Returns are a function of data. Then, by definition, returns are a statistic. As such, it is mathematically improper to assume their distribution into existence. You must not assume that $R_t$ is lognormally distributed as it must be derived.

Second, the $-1$ part of the initial formula is irrelevant to the stochastic one, so $R_t$ is a product distribution of two ratio distributions. I do apologize, but I won’t complete the math here as it is too long, but I will cover it enough in general to get a good solution for you.

For simplicity, we will ignore liquidity as Markowitz did, though that is very improper because it will violate the Dutch Book Theorem. Nonetheless, if we do not care that our formulas are not useable, we will pretend that $P=P$ instead of $P=P(Q)$. We will also abolish dividends, again, the same problem as above.

Suppose we ignore short selling for this discussion as it is nearly symmetric, “nearly” being a big deal, in reality. In that case, we can focus on $q_{t+1},$ $p_t$, and $p_{t+1}$ to get to the crux of the matter. The problem with $q_{t+1}$, even if we ignore liquidity costs is that a bankruptcy court may set it to $q_{t+1}=0$ , another firm may substitute its shares in a merger, or cash may take its place in a cash for stock merger. So our distribution for the volume ratio will be multinomial even without considering liquidity costs.

As such, the probability of a return will be the weighted sum over the possible future states. We will consider only the case where the firm remains a going concern at the end of the period. There can be no mergers, dividends or bankruptcy and we live in a sea of infinite liquidity.

Relaxing those constraints does not alter the problem but does make it very long.

So we can restrict our case to $$R_t=\frac{p_{t+1}}{p_t}.$$ The distribution of prices will depend on the auction rules. For example, in an English style auction, there is a winner’s curse. The high bidder wins, so the distribution of winning bids should be the Gumbel distribution. Only winning bids get recorded as the market price. The others represent transactions that did not happen.

In a NYSE style double auction, buyers bid against buyers and sellers bid against sellers. In equilibrium, there is no winner’s curse. The rational behavior is for each bidder to bid their expectation. As such, the limit book as time goes to infinity should be a distribution of expectations around an equilibrium price. From the central limit theorem, once scaled, it should be normally distributed.

There are two ways to handle this distribution. The first and most direct way is simply to solve $$R_t=\frac{p_{t+1}}{p_t}.$$ The problem with this is that for most of history, we only have end of day values and intraday trades are not recorded in the order they happen in due to larger orders being taken off the tape and reinserted later. An alternative exists by converting to polar coordinates around the equilibrium. In that case, we end up with one less parameter to estimate and a workable solution.

It is well known in the statistical literature that the distribution of two normal distributions has no first or higher moment, so the even moments turn out to be infinite and the odd moments do not exist. The entire distribution is a weighted sum over the states, but as long as one of the components of the sum has infinite variance, then the entire distribution has infinite variance.

A proof for a simple case exists on the mathematical dictionary at Wolfram research. The link is here.

The distribution must be truncated at 0. As an empirical note, once you have factored in all the other things and structural breaks, you end up with an excellent model of the real world.

A broader, more general discussion can be found at

Marsaglia, G. (2006). Ratios of Normal Variables. Journal of Statistical Software, 16(4), 1–10 or Marsaglia G (1965). Ratios of Normal Variables and Ratios of Sums of Uniform Variables. Journal of the American Statistical Association, 60, 193–204.

Now let us talk about what happens when you map variables without a mean onto variables without a mean.

The method of ordinary least squares does not have any distributional assumptions. That does not imply that it works as intended for all possible distributions, hence the restriction to finite variance.

Ordinary least squares is a projection and an algorithm. Indeed, all Frequentist decision rules are algorithms. If the assumptions are violated, no matter how badly, they will still pop out a number. The number can be meaningless, but it will exist. It will minimize the squared loss that would have happened at the time the data was collected had the data been known in advance of collecting it. The regression line will be the best fit line in terms of minimizing squared loss in the observed sample.

In 1851, the mathematician Augustin Cauchy entered into a battle with the mathematician Irenee-Jules Bienayme. You probably have never heard of Bienayme, but his work is everywhere in statistics, but is always named for someone else. Chebychev’s theorem was solved by Bienayme, for example.

What Bienayme basically showed in 1851 was that OLS was BLUE. As entire fields of mathematics were discovered by Cauchy, he took this as a personal insult as he had just published a median based regression method. Augustin Cauchy was probably in the top ten mathematicians of all time.

What Cauchy discovered was that if there was infinite variance, then OLS would always fail to produce a useful answer. In fact, there is no mathematical difference between having two pairs of coordinates and one million pairs of coordinates. Adding data does not improve the quality of the regression. Any form of least squares regression has been known since 1851 to produce spurious results with variables that lack a finite second moment.

I have found a contracted and simplified proof of that for the univariate case that I will link at the end.

To understand why this is happening, think about what squared loss is trying to do, it is trying to minimize the variance of an estimator that has infinite population variance. How precisely do you minimize infinity?

A proof of what is happening is too long to present here but is present in standard statistical texts that are not for applied use. Some intuition can be provided, however.

A peculiar property of distributions with infinite variance is that the sample variance grows with the sample size. That makes sense because as the sample size goes to infinity, the observed variance converges to the true variance, which is infinite. For the least-squares estimator, the denominator goes to infinity as the sample size grows. A discussion of the numerator is more complex because a covariance cannot exist, but an analogous concept exists.

What ends up happening is that the sampling distribution of the estimators eventually map out to the population distribution of the set of all possible slopes. That makes inference meaningless and the sample estimates pointless.

What you can do is one of three things.

If you are doing academic work, then you can use either Thiel’s regression or Quantile regression because all distributions have a median. Thiel’s regression is the better of the two in terms of efficiency. It is related to bootstrapping methods but is very slow. Quantile regression can have more problems with outliers. Although the breakpoint is very high, it is not infinite.

If you are doing applied work, then you must use Bayesian regression. There does not exist a sufficient point statistic for this type of regression, but the Bayesian likelihood function is always minimally sufficient. There is simply too large of a qualitative difference between the Bayesian and Frequentist methods in that case. Plus, Frequentist methods violate the Dutch Book Theorem. As such, it is possible to force banks to take losses if users use Bayesian methods and banks use Frequentist methods.

A rather bizarre element of mapping distributions lacking second moments onto distributions lacking second moments, the OLS estimator will meet all standard Frequentist criteria used in econometrics for validity. It will have the bizarre property, however, of having zero precision. The estimator will be perfectly imprecise at estimating the population quantity as the sample size goes to infinity.

Los Alamos National Labs produced a detailed discussion of this issue at

Hanson K.M., Wolf D.R. (1996) Estimators for the Cauchy Distribution. In: Heidbreder G.R. (eds) Maximum Entropy and Bayesian Methods. Fundamental Theories of Physics (An International Book Series on The Fundamental Theories of Physics: Their Clarification, Development and Application), vol 62. Springer, Dordrecht. https://doi.org/10.1007/978-94-015-8729-7_20

This type of problem also appears in particle physics and in the physics of rolling objects.

A univariate case, where there is no regression, is Gull’s Lighthouse problem. If you imagine a rotation of the regression line and a collapsing of the errors onto a single variable, then the projection of the rotated variables ends up as this univariate case. The point of projection of the light of a lighthouse has this property.

A slide presentation of it is at Imperial college. It is mostly 13 pages of equations here.

The original statement of the problem is at

Gull, Stephen (1988). Bayesian Inductive Inference and Maximum Entropy. Maximum Entropy and Bayesian Methods in Science and Engineering. Vol 1. 53-74.

$\endgroup$
14
  • $\begingroup$ Thank you for this very detailed discussion. My takeaway from it is that OLS will be unbiased but inconsistent if there is an infinite variance regressor. Is that correct? $\endgroup$ Dec 21, 2021 at 8:38
  • $\begingroup$ So our distribution for the volume ratio will be multinomial: multinomial? Also, very interesting historical notes! What is their source (not only in this, but in your other posts, too)? $\endgroup$ Dec 21, 2021 at 13:54
  • $\begingroup$ @MichaelGmeiner yes. With an infinite amount of data the closest solution that you will get is $\theta\in\Re$. It was first discovered by Poisson. It rarely gets discussed because Poisson noted, that we will doubtlessly not encounter that case. That translated quote still appears every now and then. I am sure most people do not notice it. He recommends a footnote as it is irrelevant to reality. There are exceptions, but not to OLS. Theil's regression, logit or Bayesian methods do not necessarily have this issue. $\endgroup$ Dec 23, 2021 at 3:15
  • $\begingroup$ @RichardHardy Multinomial only if $P=P$ and not if $P=P(Q)$. Even then, you could alter that model by transforming a logistic regression so that you have that $\hat{p}_i$ being modeled, which I have done. It works wonderfully. I have a wonderful bankruptcy model. I have a so-so merger model. It still has too much sampling variance. It may be nature is the issue, however, and not the model. $\endgroup$ Dec 23, 2021 at 3:20
  • $\begingroup$ @RichardHardy this is not the place for chat, but I will try and give a brief background. I used to manage money and someone noticed that my portfolio was about a six month leading signal for Berkshire Hathaway's purchases. I had some deaths and other things and I decided that I would figure out why Modern Portfolio Theory doesn't work. I am not generally bright enough to really notice how stupid such a quest was. So I read the bibliography section of every article that was relevant and traced them back to articles as far back as Maria Agnesi in the 18th century. $\endgroup$ Dec 23, 2021 at 3:25
1
$\begingroup$

There is no problem with infinite variance regressors. The assumption is made because it results in consistency at the rate of the square root of the sample size. This is standard and common, thus it is presented in this way in most textbooks.

In the case of an infinite variance regressor, the OLS estimate converges to the truth at a rate faster than $\sqrt{N}$, and is "superconsistent".

Consider the model: $y_i =\beta_0 +\beta_1 x_{1i}+u_i$.

The OLS estimator of $\beta_1$ is $$\hat{\beta_1} =\beta_1 \frac{Var(x)}{Var(x)} + \frac{Cov(x,u)}{Var(x)}$$

Consider the case of a mean-zero regressor for convenience: $$\hat{\beta_1} =\beta_1 +\frac{\frac{1}{N}\sum_{i=1}^Nx_iu_i}{\frac{1}{N}\sum_{i=1}^Nx_i^2} $$

$$\sqrt{N}(\hat{\beta_1} -\beta_1)=\frac{\frac{1}{\sqrt{N}}\sum_{i=1}^Nx_iu_i}{\frac{1}{N}\sum_{i=1}^Nx_i^2} $$

If a CLT applies to the numerator and the LLN applies to the denominator (i.e., the regressor has finite variance), then the OLS estimate converges at the rate of $\sqrt{N}$. If there is an infinite variance regressor, then the denominator in the above expression does not converge. Suppose however, that $\frac{1}{N^2}\sum_{i=1}^Nx_i^2$ converges to a non-zero constant, then,

$$N^{3/2}(\hat{\beta_1} -\beta_1)=\frac{\frac{1}{\sqrt{N}}\sum_{i=1}^Nx_iu_i}{\frac{1}{N^2}\sum_{i=1}^Nx_i^2} $$

Again, a CLT applies to the numerator, and the denominator converges. Thus, the OLS estimate converges at the rate $N^{3/2}$. This is "superconsistency" and a case that most basic textbooks rule out for ease of exposition.

$\endgroup$
4
  • $\begingroup$ Do you have any textbooks or extended reading on this topic? I note that this answer has much less divergence - I am not interested in finance per se so it is quite clear. $\endgroup$ Jan 31 at 16:08
  • 1
    $\begingroup$ Hamilton's Time Series Analysis, section “Asymptotic Properties of a First-Order Autoregression when the True Coefficient is Unity” (17.4). $\endgroup$ Jan 31 at 16:21
  • $\begingroup$ This is most prominent in time series econometrics in which regressors have unit roots or are otherwise non-stationary, giving them infinite variance. $\endgroup$ Jan 31 at 16:21
  • $\begingroup$ Ah, I should have known better than to think it would be anywhere else! $\endgroup$ Jan 31 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.