I'm third year economics student and all econometrics we had so far and basically all empirical studies in economic subjects we had so far are linear regression. Is there any alternative, can anyone suggest any reading material or direction in which I can explore?
4
-
$\begingroup$ It strongly depends on what you want to do: forecasting? classification? $\endgroup$– caveracCommented Nov 25, 2017 at 11:24
-
$\begingroup$ Can you give me example for both? $\endgroup$– econCommented Nov 25, 2017 at 13:37
-
$\begingroup$ By linear regression, do you mean linear in the parameters or linear functional form? There is an explanation of the difference at blog.minitab.com/blog/adventures-in-statistics-2/… $\endgroup$– Adam BaileyCommented Nov 25, 2017 at 17:03
-
$\begingroup$ Linear functional form. $\endgroup$– econCommented Nov 25, 2017 at 20:35
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
Regression is the mapping of any function of any dimension onto a result. There exists an infinite number of functions. Furthermore, there is a wider range of linear regression tools than just least squares style solutions. My guess is that you have yet to even come close to covering the linear tools if you are in your third year.
For more linear tools, look at quantile regression and Theil's regression. Both are very robust. Quantile, ordinary least squares and Theil's method of regression are usable for a polynomial of any degree. If you are studying least squares style methods such as OLS, GLS or FGLS then you are also studying non-linear methods at the same time. All linear tools can be readily adapted to non-linear problems. The part no one has told you is that you are studying non-linear regression, but using linear models to study the properties because they are easier.
The linkage is similar to the linkage between Algebra 2 in high school and calculus 1. Grounding in the former is necessary for the latter.
Instead of worrying about "non-linear" methods, I would recommend taking two different routes.
The first is non-parametric and distribution-free methods. The second is Bayesian methods. Your instructor will hate me forever for this recommendation.
Distribution-free methods are the easiest to understand. They are robust under any distributional assumption, which results in a material loss of power. They always work, but they are weak solutions because you know so little about how the world works. Theil's regression is such an example.
Non-parametric methods are a little harder to understand. They do not depend upon a parameter to perform inference. So, for example, when you have performed a t-test, you have assumed that a mean exists and that it is meaningful. It is not always true that a mean exists in a distribution and it is not always true that it is a meaningful measurement when it does exist. Non-parametric methods allow you to perform tests on data without reference to a parameter. As with distribution-free methods, they are weaker than the equivalent parametric test. They always work, but they are more likely to not detect an effect that is really there.
Finally, after you have looked at distribution-free and non-parametric methods, then you should look at Bayesian methods. Bayesian methods are older than Frequentist methods, but allow you to solve problems that have no Frequentist solution. On the surface, they can look just like the problems you are solving now, but below the surface, they open up entire world's of prediction and modeling not available with Frequentist methods.
Bayesian methods reverse the direction of uncertainty. With a null hypothesis method, you assert that the null is true and use the data to falsify it. Essentially you are performing a test as $\Pr(x|\theta)$, that is to say, what is the probability of seeing this data if the null is actually true. The use of Bayesian methods reverses the question. The Bayesian asks $\Pr(\theta|x)$? The Bayesian method asks, "what is the probability the hypothesis is true, given the data that was actually seen?"
The Frequentist works in the "sample space," which is the set of all possible outcomes of a random event. The Bayesian works in the "parameter space" which is the set of all possible explanations.
A good post showing the differences that you can readily see is on the difference between a Frequentist confidence interval and a Bayesian credible interval. It is at https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval
William Bolstad writes a good introductory book on Bayesian methods if you have had calculus through integration. You cannot do Bayesian methods without knowing integration.
There is a giant world out there. Go explore.
$\begingroup$
$\endgroup$
2
There are numerous directions to go which start moving you beyond ordinary least squares (OLS), linear regression. The universe of statistical methods is large!
Two books that I particularly enjoyed are Econometrics by Hayashi and Elements of Statistical Learning by Hastie et. al. Looking back at your question, these books may be too advanced. But maybe not. An easier version of the latter is An Introduction to Statistical Learning (and may be interesting from the perspective of a broader exposure to data science than just econometrics).
- Hayashi's Econometrics introduces a variety of methods through the lens of GMM and with an eye towards time-series econometrics.
- Elements of Statistical Learning is a modern classic of the statistics, machine learning literature. It's great for opening your eyes to methods outside of traditional econometrics.
Some examples beyond ordinary least squares...
Maximum likelihood estimation (MLE)
You must know this if you continue doing statistics. It's a ubiquitous workhorse.
If you can specify the likelihood function then parameters of the likelihood function can be estimated by maximizing the likelihood function. In certain special cases, (eg. linear regression with conditionally normal error terms) the OLS estimator is the MLE estimator. You've undoubtedly encountered MLE estimation before if you've estimated a logit model. MLE is all over physics, engineering, and the sciences.
There are issues with applying MLE to economic models though. Often we know that an overall economic model is false. A model will imply certain facts that are entirely fallacious. Forcing the model to match the data in a maximum likelihood sense may not induce a useful choice of parameters. GMM can be used as an alternative approach to selectively test certain predictions of a model.
GMM is another broad method for estimating parameters based upon moment conditions that in expectation should be zero. Hayashi's book Econometrics develops ordinary least squares regression, instrumental variables, maximum likelihood, and other methods as special cases of GMM with different moment conditions. OLS can be thought of as GMM using the orthogonality condition of the regressors and the error terms. MLE can be derived as GMM on the score.
Matching methods for estimating causal effects are common in certain areas of economics.
The idea is to match a treated entity with an untreated entity based upon observable characteristics. A widely used technique for example is propensity score matching
There are all kinds of variations on classic linear methods:
The idea here is to start with ordinary least squares but then to bias coefficient estimates towards zero to reduce overfitting and improve out of sample prediction.
References
Fumio, Hayashi, 2000, Econometrics
Hastie, Trevor, Robert Tibshirani, Jerome Friedman, 2009, Elements of Statistical Learning
-
$\begingroup$ Thanks! I read Introduction to statistical learning and mainly motivation for this question came from disappointment of not seeing any regularization methods used in our economics and econometrics classes...thanks :) $\endgroup$– econCommented Dec 3, 2017 at 11:03
-
$\begingroup$ @econ You're going to see more machine learning techniques used in economics, especially for problems or subproblems where the interest is in a forecast $\hat{y}$ and you don't care as much why or how you got that forecast. Be aware though that many machine learning techniques can be problematic for $\hat{\beta}$ problems where you're trying to estimate a specific parameter (eg. what's the causal effect of $x$). Machine learning techniques etc... may also provoke hostile questions from many economists who don't know the techniques and/or view them with suspicion. $\endgroup$ Commented Dec 4, 2017 at 18:00
$\begingroup$
$\endgroup$
Linear regression, despite its simplicity, it is actually a very powerful tool. That's why it's everywhere in econometrics, to give you an example you're maybe familiar with, consider an auto-regressive model, turns out you can write the future state of a variable that follows this model as a linear combination of previous states
$$ X_t = C + \phi_1 X_{t-1} + \phi_2 X_{t-2} + \cdots + \phi_p X_{t-p} + \eta_t \tag{1} $$
so, if you know the weights $\{\phi_k \}_{k=1}^p$ you can predict what's going to be the future values of $X$. The interesting part is that these numbers are obtained through linear regression: just call $y = X_t$, $x_1 = X_{t-1}, x_2 = X_{t-2},\cdots$ and realize Eq. (1) can be written as
$$ y = C + \phi_1 x_1 + \phi_2 x_2 + \cdots + \phi_p x_p + \epsilon \tag{2} $$
In this sense linear regression can be used for forecasting. But there are other tools, I will link you to this other thread where you can get an idea on how neuronal networks can be used for this task: forecasting time series. But forecasting can also be done with a plethora of methods: support vector machines are popular choices.
$\begingroup$
$\endgroup$
You can always think of a linear regression model as a first-order Taylor approximation of some nonlinear regression model. So linearity itself is not a particularly serious issue.
That being said, there are a couple of common nonlinear regression models that are accessible at the undergraduate level, namely, models with binary dependent variables: logit and probit.
