I know that a regularized logistic regression can be done to reduce training error, but I haven't found any economics research that uses a regularized probit model, only a regular probit model from what I have been able to find. Why don't economic researchers use a regularized probit model?

Thank you

  • 1
    $\begingroup$ It's all related with the purpose of regularization. Regularization is used to reduce test error (not training error) for better prediction. For prediction, logit and probit are not much different, and logit is more popular and easier. Not much point pursuing probit; it's too minor. Causality is different as it is strictly based on model. $\endgroup$ – chan1142 Jan 12 '20 at 6:08
  • $\begingroup$ Yes, regularization is used to reduce test error, I misspoke, sorry. Thanks for your reply! $\endgroup$ – John Balzani Jan 13 '20 at 13:30
  • $\begingroup$ Right. I also thought you just mistyped it. BTW I liked your thoughtful question. $\endgroup$ – chan1142 Jan 13 '20 at 15:04

I have spent three days thinking about how to answer your question. I still haven't really decided how it should be answered, so this answer is more in the way of a set of observations. I can readily imagine a formal treatment of general linear models or limited dependent variables models could do better than this answer.

The first thing I would observe is there is probably no answer to your question. Instead, you should consider constructing such a solution. It is probably impossible to talk about why something did not happen.

The second is that every academic tradition tends to have traditions about what and how to use a statistical model. There is a lot of discretion in a pretty wide range of problems.

Some of that tradition is influenced by computability and the size of data sets. Many fields have been heavily influenced by the role punch-card computing had on a choice of solutions. For older fields, where tables or slide rules were used, you see its influence. At least some of the question is likely due to a field feeling the need to explore some methods more than others.

Other parts are influenced by the frequency of some type of problem. If a field commonly runs into censoring, for example, it still influences the thought even when censoring isn't present. Economists generally cannot perform randomized experiments and are being asked by policymakers to make decisions almost for them. The economics profession loves to assume normality, even when it cannot be present, because it makes for tractable solutions.

Another observation is that all regularization methods map to some Bayesian point estimator under highly stylized assumptions. That is not trivial because it biases the estimator. The small loss in accuracy can lead to a substantial improvement in precision.

That would matter more for sampling distributions that have heavier tails than others. The mean of the normal distribution has better sampling properties than the mean of the logistic distribution. To see this, I created ten thousand samples of size one thousand and found the mean of each sample and plotted its density plot. Each sample has a mean of zero and a variance of one.

The density plot of the means of the normal distribution are in red and the logistic are in blue.


The purpose of regularization is to make the leptokurtic sampling distribution of the mean for the logistic distribution become closer to the mesokurtic distribution of the mean of the normal distribution.

That, of course, can be the side effect of a well constructed Bayesian prior distribution. Indeed, it is one of the reasons Bayesian models are less likely to overfit. Of course, they have their own unique problems that Frequency-based models lack.

Each of these densities is the "test" distribution given a known distribution. Because the slope of the regression, or the mean here, is easily pulled away from the true parameter value when compared to the normal mean you can reduce the width of the test distribution by not weighting all values on the real number line as equiprobable.

So one of the likely answers would be that the advantage of regularization would produce smaller gains for probit. Nonetheless, Stein's paradox would still hold, though in a strange form, and there would be gains.

Indeed, the entire point of Bayesian estimation is to impute real outside knowledge into the estimator. If something is unexpected, then the weighting implicit in a Bayesian estimate pulls any point estimator away from the sample on the logic it is an outlying sample. Of course, the potential conceit is that the researcher would be able to identify a sample that is unusual.

Regularization automates that conceit without asking the researcher if they do have actual knowledge from prior research.

There is no reason not to find a regularization for higher dimensional probit, but there is a need to do it with a high dimension logit.

It is certainly a feasible thing to do because Bayesian priors for logistic regression are discussed quite a bit in the literature. Finding one that works well so that an analytic solution accepted by your peers can be found is quite another issue.

  • $\begingroup$ Thank you for the detailed answer! That answers my question. $\endgroup$ – John Balzani Jan 17 '20 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.