So let us begin with a real case where the variance has to be infinite, the stock market.
I am not doing a complete derivation here as it would run as multiple chapters. I know this because I am doing web presentations on this. But I can do enough to show you that there cannot be a first moment for the stock market. From that, I will explain the consequence for regressors.
Let us begin with a couple of simple observations about a return or a reward for investing.
If we define a return on a single pair of cash flows as $$r_t=\frac{p_{t+1}}{p_t}\times\frac{q_{t+1}}{q_t}-1$$ and $$R_t=r_t+1$$ we can note a few things.
First, $$R_t=R(p_t,p_{t+1},q_t,q_{t+1}).$$ By definition, prices are data. Volumes are data. Returns are a function of data. Then, by definition, returns are a statistic. As such, it is mathematically improper to assume their distribution into existence. You must not assume that $R_t$ is lognormally distributed as it must be derived.
Second, the $-1$ part of the initial formula is irrelevant to the stochastic one, so $R_t$ is a product distribution of two ratio distributions. I do apologize, but I won’t complete the math here as it is too long, but I will cover it enough in general to get a good solution for you.
For simplicity, we will ignore liquidity as Markowitz did, though that is very improper because it will violate the Dutch Book Theorem. Nonetheless, if we do not care that our formulas are not useable, we will pretend that $P=P$ instead of $P=P(Q)$. We will also abolish dividends, again, the same problem as above.
Suppose we ignore short selling for this discussion as it is nearly symmetric, “nearly” being a big deal, in reality. In that case, we can focus on $q_{t+1},$ $p_t$, and $p_{t+1}$ to get to the crux of the matter. The problem with $q_{t+1}$, even if we ignore liquidity costs is that a bankruptcy court may set it to $q_{t+1}=0$ , another firm may substitute its shares in a merger, or cash may take its place in a cash for stock merger. So our distribution for the volume ratio will be multinomial even without considering liquidity costs.
As such, the probability of a return will be the weighted sum over the possible future states. We will consider only the case where the firm remains a going concern at the end of the period. There can be no mergers, dividends or bankruptcy and we live in a sea of infinite liquidity.
Relaxing those constraints does not alter the problem but does make it very long.
So we can restrict our case to $$R_t=\frac{p_{t+1}}{p_t}.$$ The distribution of prices will depend on the auction rules. For example, in an English style auction, there is a winner’s curse. The high bidder wins, so the distribution of winning bids should be the Gumbel distribution. Only winning bids get recorded as the market price. The others represent transactions that did not happen.
In a NYSE style double auction, buyers bid against buyers and sellers bid against sellers. In equilibrium, there is no winner’s curse. The rational behavior is for each bidder to bid their expectation. As such, the limit book as time goes to infinity should be a distribution of expectations around an equilibrium price. From the central limit theorem, once scaled, it should be normally distributed.
There are two ways to handle this distribution. The first and most direct way is simply to solve $$R_t=\frac{p_{t+1}}{p_t}.$$ The problem with this is that for most of history, we only have end of day values and intraday trades are not recorded in the order they happen in due to larger orders being taken off the tape and reinserted later. An alternative exists by converting to polar coordinates around the equilibrium. In that case, we end up with one less parameter to estimate and a workable solution.
It is well known in the statistical literature that the distribution of two normal distributions has no first or higher moment, so the even moments turn out to be infinite and the odd moments do not exist. The entire distribution is a weighted sum over the states, but as long as one of the components of the sum has infinite variance, then the entire distribution has infinite variance.
A proof for a simple case exists on the mathematical dictionary at Wolfram research. The link is here.
The distribution must be truncated at 0. As an empirical note, once you have factored in all the other things and structural breaks, you end up with an excellent model of the real world.
A broader, more general discussion can be found at
Marsaglia, G. (2006). Ratios of Normal Variables. Journal of Statistical Software, 16(4), 1–10 or Marsaglia G (1965). Ratios of Normal Variables and Ratios of Sums of Uniform Variables. Journal of the American Statistical Association, 60, 193–204.
Now let us talk about what happens when you map variables without a mean onto variables without a mean.
The method of ordinary least squares does not have any distributional assumptions. That does not imply that it works as intended for all possible distributions, hence the restriction to finite variance.
Ordinary least squares is a projection and an algorithm. Indeed, all Frequentist decision rules are algorithms. If the assumptions are violated, no matter how badly, they will still pop out a number. The number can be meaningless, but it will exist. It will minimize the squared loss that would have happened at the time the data was collected had the data been known in advance of collecting it. The regression line will be the best fit line in terms of minimizing squared loss in the observed sample.
In 1851, the mathematician Augustin Cauchy entered into a battle with the mathematician Irenee-Jules Bienayme. You probably have never heard of Bienayme, but his work is everywhere in statistics, but is always named for someone else. Chebychev’s theorem was solved by Bienayme, for example.
What Bienayme basically showed in 1851 was that OLS was BLUE. As entire fields of mathematics were discovered by Cauchy, he took this as a personal insult as he had just published a median based regression method. Augustin Cauchy was probably in the top ten mathematicians of all time.
What Cauchy discovered was that if there was infinite variance, then OLS would always fail to produce a useful answer. In fact, there is no mathematical difference between having two pairs of coordinates and one million pairs of coordinates. Adding data does not improve the quality of the regression. Any form of least squares regression has been known since 1851 to produce spurious results with variables that lack a finite second moment.
I have found a contracted and simplified proof of that for the univariate case that I will link at the end.
To understand why this is happening, think about what squared loss is trying to do, it is trying to minimize the variance of an estimator that has infinite population variance. How precisely do you minimize infinity?
A proof of what is happening is too long to present here but is present in standard statistical texts that are not for applied use. Some intuition can be provided, however.
A peculiar property of distributions with infinite variance is that the sample variance grows with the sample size. That makes sense because as the sample size goes to infinity, the observed variance converges to the true variance, which is infinite. For the least-squares estimator, the denominator goes to infinity as the sample size grows. A discussion of the numerator is more complex because a covariance cannot exist, but an analogous concept exists.
What ends up happening is that the sampling distribution of the estimators eventually map out to the population distribution of the set of all possible slopes. That makes inference meaningless and the sample estimates pointless.
What you can do is one of three things.
If you are doing academic work, then you can use either Thiel’s regression or Quantile regression because all distributions have a median. Thiel’s regression is the better of the two in terms of efficiency. It is related to bootstrapping methods but is very slow. Quantile regression can have more problems with outliers. Although the breakpoint is very high, it is not infinite.
If you are doing applied work, then you must use Bayesian regression. There does not exist a sufficient point statistic for this type of regression, but the Bayesian likelihood function is always minimally sufficient. There is simply too large of a qualitative difference between the Bayesian and Frequentist methods in that case. Plus, Frequentist methods violate the Dutch Book Theorem. As such, it is possible to force banks to take losses if users use Bayesian methods and banks use Frequentist methods.
A rather bizarre element of mapping distributions lacking second moments onto distributions lacking second moments, the OLS estimator will meet all standard Frequentist criteria used in econometrics for validity. It will have the bizarre property, however, of having zero precision. The estimator will be perfectly imprecise at estimating the population quantity as the sample size goes to infinity.
Los Alamos National Labs produced a detailed discussion of this issue at
Hanson K.M., Wolf D.R. (1996) Estimators for the Cauchy Distribution. In: Heidbreder G.R. (eds) Maximum Entropy and Bayesian Methods. Fundamental Theories of Physics (An International Book Series on The Fundamental Theories of Physics: Their Clarification, Development and Application), vol 62. Springer, Dordrecht. https://doi.org/10.1007/978-94-015-8729-7_20
This type of problem also appears in particle physics and in the physics of rolling objects.
A univariate case, where there is no regression, is Gull’s Lighthouse problem. If you imagine a rotation of the regression line and a collapsing of the errors onto a single variable, then the projection of the rotated variables ends up as this univariate case. The point of projection of the light of a lighthouse has this property.
A slide presentation of it is at Imperial college. It is mostly 13 pages of equations here.
The original statement of the problem is at
Gull, Stephen (1988). Bayesian Inductive Inference and Maximum Entropy. Maximum Entropy and Bayesian Methods in Science and Engineering. Vol 1. 53-74.
