# What should we conclude when the significant levels are different but the sign are similar among coefficients of variable of interest?

I saw the impact of anticollusion laws on dependent variables Y across the country by using generalized DID by following Dasgupta, 2019.

The identification is:

$$Y_{it}$$ = $$\alpha$$ + $$\beta$$ $$(pt)_{kt}$$ + $$\deltaX_{ikt}$$ + $$\theta_t$$ + $$\gamma_i$$ +$$\epsilon_{it}$$

where i,k, and t index firms, countries, and years respectively. $$X_{ikt}$$ is a vector of the different firm, country, and industry control, while $$\gamma$$ and $$\theta$$ are firm and year fixed effects.$$(pt)_{kt}$$ is the post * treat variable

The result is

While 6 columns all using firm and years fixed effects if not stated elsewhere.Column 1, I did not control any independent variables. Column(2), I control for some firm and country independent variables. In column (3), I control for firm, country and industry variables. in column (4), I control for country and firm independent variable along with firm and industry * year fixed effect. In column (5). I control for the country and firm independent variables along with firm and region * year fixed effect. In column (6), I control for some firm and country independent variables, similar to column (2) but without US firms.

I am wondering whether I can conclude that: anticollusion laws, in general, have weak but consistent negative impact on Y ceteris paribus, on average in this situation?

I am wondering whether I can conclude that: anticollusion laws, in general, have weak but consistent negative impact on Y ceteris paribus, on average in this situation?

I think that would be too strong wording in this case.

Here the results are not very consistent or robust since in the model 1 the effect is positive and in model 2-4 the effect is not statistically significant so you should there you cannot reject the null hypothesis of true effect being 0. You should not interpret results in 2-4 as saying the effect is negative (well for 2 you could at 10% level). So you can only really say there is a significant negative effect in 2 (3 at 10%) models, and even then the magnitude of the effect is not necessarily consistent.

However, when it comes to the magnitude of the effects, it is hard to comment on the magnitude of the coefficients since you do not say what $$Y$$ is or how is it measured, ignoring the first result with no controls, the magnitude of a coefficient in model 5 is 35,7% higher than the magnitude of the coefficient in model 6. Depending on how $$Y$$ is measured this might be high difference. For example, if $$Y$$ is output in billions that would a large difference in effect, if it is output in dollars that is difference of just few cents. Asking whether effect here is weak or strong in economic sense is essentially asking whether 0.02 is high number... well if its 0.02 billions or trillions of something then it could be a strong effect.

• Hi @1muflon1, sorry for this unclear, the dependent variable here is the actual number from the ratios. For example, if dependent variable is 2%, I will let the value is 0.02 in this case and run the regression. Please let me know if it is still not clear. Jun 16, 2021 at 20:39
• @Knowledge-chaser ok but ratio of what if 0.02 is return 2% higher/lower ROI that is actually quite a large effect, maybe not super strong, but 2% increase/decrease in ROI is nothing to scoff at, but for some other variable 2% change might be nothing
– 1muflon1
Jun 16, 2021 at 20:44
• @Knowledge-chaser people typically include multiple models to see how robust coefficients are to addition of controls, even though the most parsimonious model without any controls will likely suffer from OVB its still often done, I don't see any issue with that but also I would definitely not focus on coefficient reported in first regression. As to what explains the change in point estimates its likely the omitted variable bias (OVB) mentioned above
– 1muflon1
Jun 16, 2021 at 21:20
• @Knowledge-chaser please read more carefully what I have written, I did not say that there is a difference of 35 percent, I stated the coefficient has 34% higher magnitude since $-0.019/-0.014\approx 1.357$
– 1muflon1
Jun 16, 2021 at 21:27
• @Knowledge-chaser no I do not think that would be correct interpretion as well here too. What I would have written here is that, the effect of the anti-collision laws is not robust. Most general models report statistically significant negative effect but coefficient magnitude is not robust either (this last part after but still depends on what even Y is since you did not mentioned it in the comments). More parsimonious models are either not significant or report significant positive effect. After that you can add some arguments why you think that higher weight should be put on last 2 results
– 1muflon1
Jun 16, 2021 at 21:30