# Is my instrument too noisy?

I have an instrument, that I want to use with IV 2SLS, to predict my endogenous variable.

The scatter plot of my instrument against my endogenous variable looks like this.

Using Stata's binscatter, I can absorb control variables and fixed effects, and the resulting binscatter plot looks like this. In the X axis, all data points are distributed across 100 equal bins, and the chart reports the mean value for each bin, for values in the X and Y axes (after controlling for control variables and fixed effects).

The First Stage of IV 2SLS is statistically significant (0.357***, using robust standard errors), and the F Stat is above the 10 critical threshold.

My question is: is instrument too noisy? I see many outliers, and I am not sure about whether the fact that I get a positive and significant First Stage, and F Stats above 10, is a statistical artefact. I guess I am wondering if the data distribution in my scatter plot makes it unappropriate for OLS and for these relevance tests.

Thanks!

• The binscatter looks great and you pass the F-Test rule of thumb, I don't see why you worry? Noisiness would reduce instrument power, to further convince readers that this is not the case you could use weakiv' test (in Stata). Commented Aug 31, 2022 at 11:36
• Thanks @Papayapap. I guess I worried because the scatter plot does not look clear to the eye. I guess it is a matter of it having 2,800 points and the eye can't tell much.
– Xavi
Commented Aug 31, 2022 at 16:12
• Coming from a biology background, why don't you use a log scale on both X and Y axis, in your first graph? Commented Jan 30, 2023 at 20:17

I would advise against trying to determine if something is outlier or not just visually. Even visually I would say there are only 2 clear outliers in first graph and 1 in the second. You can try to check more rigorously by computing some metric for reach observation. For example you could use Z-score, and afterwards you can try to do robustness check by removing any observations farther than $$\pm 3\sigma$$.

Also many statisticians would caution against removing even genuine outlier, unless it’s clear mistake in entry (eg height of person of 1 cm etc). Outliers, if not results of errors, are still genuine observations that contain information about the true relationship.

Regarding whether data are noisy or not that does not depend on outliers per se but on standard errors. Higher standard errors mean more noise. However, there isn’t any rule of thumb about noisiness per se.

When it comes to noisiness in data the issue is that $$t$$-stat is calculated as $$\hat{\beta}/se(\hat{\beta})$$ (see Verbeek A Guide to Modern Econometrics). If there is a lot of noise (se is high) you are more likely to not reject null even if perhaps there is true relationship that you would be able to see in less noisy data. However, having noisy data does not make your instruments invalid.

• I am currently using standard errors robust for heteroskedasticity, in light of the shape of the scatterplot. This increases my standard errors, as expected. Yet, my First-Stage coefficient is still positive and statistically significant (with F Stat above 10). I hence understand that my t stats are already accounting for the distribution of data depicted in the scatter plot. I understand this should be the correct approach, right?
– Xavi
Commented Aug 31, 2022 at 10:40
• @Xavi yes it is correct to use robust errors although robust errors account not for distribution of data but for unequal variances of errors (that’s not the same thing)
– 1muflon1
Commented Aug 31, 2022 at 11:11
• Ah yes you are totally right! Thanks for your comments. My take away is that I should assess the relevance and strength of my instrument on the basis of the t-stats and p values that I obtain in First-Stage regressions - which do already take into account the noiseness in my data, through high standard errors.
– Xavi
Commented Aug 31, 2022 at 11:35
• @Xavi yes also you should do that by checking F stat. Also don’t forget you need to justify that instruments are exogenous - there is no good test for that you need to logically justify that
– 1muflon1
Commented Aug 31, 2022 at 11:41

The noisy instrument is related to the concept of relevant instrument, and traditionally this is only related to the statistical significance of the IV and the endogenous regressors. but with the statistically significant test of relevance that you obtained. You could argue that the instrument, in fact, contains explanatory power to explain the endogenous regressor.

However, given the study of Bazzi & Clemens (2013), and the concept of "relative bias" that you can encounter when using a sufficiently overidentified system, you can inspect how much of the parameter can change whenever we get noisy instruments. This is not easy to perform and, for example, can only be achieved if you got more instruments than endogenous variables. An example of this is the relative bias statistic from Stata output after using ivregress and applying the post-estimation command estat firststage`.

Ref.

Bazzi, S., & Clemens, M. A. (2013). Blunt Instruments: Avoiding Common Pitfalls in Identifying the Causes of Economic Growth. American Economic Journal: Macroeconomics, 5(2), 152–186. https://doi.org/10.1257/mac.5.2.152