# How to interpret “desired significance level”?

I was studying threats to internal validity from Stock and Watson's Introduction to Econometrics 2nd edition, chapter 14 Assessing studies based on multiple regression.

The textbook says "hypothesis tests should have the desired significance level (the actual rejection rate of the test under the null hypothesis should equal its desired significance level). and confidence intervals should have the desired confidence level.

How should I interpret this short paragraph?

What I think it means is that the hypothesis tests should have appropriate/not-too-off-tract significance level. But what is desired confidence level? And why does these conditions have to do with threats to internal validity?

Pardon my poor understanding...

I am not sure what you mean by "appropriate significance level."

Basically how sure you want to be is your choice. A frequently used analogy is the presumption of innocence. You are considered guilty only if this is proven beyond reasonable doubt. But what exactly is reasonable? This is for you to decide. If you require very strong evidence then the probability of making a false positive conclusion (Type I error) decreases. But this will increase the probability of making a false negative conclusion (Type II error). In the presumption of innocence analogy: If you change the law in a way that will require more evidence for a conviction, you will send fewer innocent people to jail (Type I), but you will also let more guilty ones go (Type II).

The significance level is the probability of a Type I error occuring.

In some situations you can perform a cost benefit analysis. In quality control you may be able to determine the cost of the damage caused by a malfunctioning part and then compare this to the cost of throwing away a properly functioning part. Using these you can actually calculate a cost minimizing significance level. However in other situations this is not feasible, or sometimes not even desirable. Then you have to make your own decision about the significance level.

In econometrics the most frequently used significance levels are 0.1%, 1% and 5%. My understanding is that this is a rule of thumb and there is actually no calculation behind it. (I could be wrong.)
Given some data you might say that "the findings are significant on the 5% level but not on the 1% level," or you can simply give the p-value, this is what is usually done.

• Hi! Really appreciate your detailed explanation! : ) So am I right to say that since the commonly used significance levels are treated as rule of thumb, then we should use them (i equate them with the"desired" significance level) in solving hypothesis problem? Otherwise, if someone adopts some weird, uncommon significance level, he or she thus introduces higher type I or II error leading to wrong conclusion. Therefore it forms a threat to internal validity? – user33448687 Apr 21 '15 at 8:07
• Not sure what you mean by internal validity? If you give the significance level then everyone (at least econometricians) can draw their own conclusions. If you claim that anything beyond level X is significant and adjust X to suit your needs that definitely tampers with the validity of your result. – Giskard Apr 21 '15 at 10:06