1
$\begingroup$

I am wondering when the sign and magnitude of coefficients of a variable of interest matter if it is statistically insignificant. Normally, I am concerning a coefficient of variable if it is significant at 5% level or lower. I am wondering when we should care about the magnitude and sign of coefficient if it is insignificant? Saying for example, my variable of interest saying about the impact of anticorruption laws on asset growth in Difference - in -Difference setting.

Please guide me to clarify my question more if it is not yet clear. Sorry @1muflon1 that I did not clarify my question that confused you

Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

answer to new edit

If the coefficient is not significant then you cannot reject the hypothesis that true coefficient is zero. In that case, magnitude or sign of the coefficient is not very relevant.

You could still care about it a bit because if you find large coefficient with sign you would expect to find, it might be that it is insignificant only because there is a lot of noise in your data (remember test statistics depends not just on coefficient size but also standard errors $\hat{\beta}/se(\hat{\beta})$. So finding large coefficient with expected sign might motivate you to perhaps find larger dataset where there is less noise, but other than this it would not be very relevant.

Answer to original question:

Of course you should care about sign and magnitude of the coefficient. This is especially true when it comes to policy analysis.

I am not familiar with the anti-corruption laws research, so let me give you another example. Consider effect of minimum wage laws on employment.

Sign of the treatment dummy clearly matters as it would be a whole world of difference if research would show that minimum wages have positive impact on employment, to case where they have negative impact on employment (which implies difficult trade-off between higher wages for low income people vs their employment).

Second, magnitude of the coefficient matters as well as again if the relationship between minimum wage and employment is such that 1% increase in minimum wage leads to 10% decrease in employment that implies the trade-off would be very severe. However, if 1% increase in minimum wages increases unemployment just by 0.0001% then no matter whether the coefficient is statistically significant or not the effect is so small it could be safely ignored and you do not even need to worry about it.

Improve this answer
$\endgroup$
8
  • $\begingroup$ I mean, when should we care about the magnitude and sign of the coefficient if it is insignificant. Sorry i did not explain my concern well. $\endgroup$
    – Nguyen Lis
    Commented Oct 12, 2021 at 21:46
  • $\begingroup$ I disagree with your point in general and have posted an alternative answer. $\endgroup$
    – Richard Hardy
    Commented Oct 23, 2021 at 13:26
  • $\begingroup$ @RichardHardy thanks for letting me know, I think your answer makes some good points, but i disagree that we should not assume that the true coefficient is zero if it’s not significant. Sure the estimations are our best guess of what the coefficient is, but if the standard errors are too high there is high chance that the coefficient is just random noise anyway I upvoted your answer because I think it makes some good points $\endgroup$
    – 1muflon1 ♦
    Commented Oct 23, 2021 at 13:30
  • $\begingroup$ I suppose the choice between the point estimate and zero has to do with prior knowledge. Without prior knowledge, what is the rationale to pick a random number (zero) over an estimate based on the data? (Zero is just as good a number as any if we lack prior knowledge.) $\endgroup$
    – Richard Hardy
    Commented Oct 23, 2021 at 13:35
  • 1
    $\begingroup$ @1muflon1, I think many a reasonable economist would not question the hypothesis that everything depends on everything (nonzero effects throughout) even if most of the effects are negligible. testing scientific theories might be a crucial bit in your comment. For that goal one is often OK with taking an unreasonable $H_0$ of zero effect (a straw man) and trying to beat it. In other words, I think there is a big difference between trying to figure out the truth and testing a scientific theory. $\endgroup$
    – Richard Hardy
    Commented Oct 23, 2021 at 14:15
3
$\begingroup$

They matter whenever you are interested in learning from the data.

If you do not have prior knowledge about the effect you are studying, the point estimate alongside its confidence interval and statistical significance tell a lot of useful things. The point estimate tells you the best guess of the effect size and its direction, as seen in the data through the lense of your model. Statistical significance puts that into perspective (how likely you are to obtain such an extreme, or even more extreme, estimated effect size if $H_0$ of zero effect is in fact true in population) but does not invalidate the fact that this is the best guess. Replacing it with a zero (which in many cases in economics, unlike genetics, is known with certainty to be incorrect) is generally not justified.

If you do have prior knowledge, you could combine it with the (new) data using e.g. Bayesian estimation. The resulting estimate would again be your best guess, and it would not make much sense to replace it with anything else (such as a zero).

This presumes you are following a sound methodology rather than trying to cheat your way to a desirable result one way or another.

Improve this answer
$\endgroup$
0
$\begingroup$

When insignificant (not significantly different from $0$), we should never interprete the sign and magnitude of the estimated coefficient. Because, with high probability the sign of the coefficient could be the opposite to the estimated one, and the magnitude of the coefficient either much smaller or much higher. The estimations are specific values taken by random variables which could be fully different for other $X$ values.
More precisely, the only interpretation which makes sense in a linear model, is that the impact of the variable $X_j$ is not significantly different from zero, the anti-corruption law is not an effective tool to fight against corruption, because in the treated sample there is comparable corruption as in the untreated sample for instance (ceteris paribus, and conditionally to the chosen model specification).

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ "high probability the sign of the coefficient could be the opposite to the estimated one" can I ask the reason why? I am wondering why it is high probability opposite to the estimated one? $\endgroup$
    – Nguyen Lis
    Commented Oct 13, 2021 at 10:03
  • 1
    $\begingroup$ Assume that $\widehat{\beta_j}=1.23$ and that the p-value is $Pr[|T|>|t||\beta_j=0]=0.37$. In this case $\beta_j=0$ cannot be rejected at the 5% threshold. This means that if the null hypothesis is true there is 37/2% chance to draw a value such that $\widehat{\beta_j}=1.23$ or higher. And as the distribution is symmetric wrt 0, there is also 37/2 % chance to draw a value lower than $\widehat{\beta_j}=-1.23$, $\endgroup$
    – Bertrand
    Commented Oct 13, 2021 at 12:59
  • $\begingroup$ -1. Your first paragraph suggests to ignore a data-based point estimate in favor of a hypothesized value under $H_0$. I do not think this is generally a smart choice. E.g. one could systematically lose money if one grounded one's bets on it, unless the $H_0$ is selected in some superior way, e.g. based on additional data. Your second paragraph seems to fall in the trap of equating absence of evidence with evidence of absence. $\endgroup$
    – Richard Hardy
    Commented Oct 23, 2021 at 13:05
  • $\begingroup$ @Richard Hardy: under $H_0: \beta_j=0$, and the data-point based evidence does not contradict this hypothesis (at the $5\%$ theshold). $\endgroup$
    – Bertrand
    Commented Oct 23, 2021 at 13:29
  • $\begingroup$ What I mean is that only interpretation <...> is that <...> the anti-corruption law is not an effective tool is a way of arguing for absence of an effect, while the statistical result merely indicates absence of (or more precisely, lack of sufficient) evidence to the contrary. We do not prove the effect is zero; we merely fail (to collect enough data) to prove otherwise. $\endgroup$
    – Richard Hardy
    Commented Oct 23, 2021 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.