Field experiments on labour market discrimination: do sizes of randomly assigned groups matter?

Question

In traditional field experiments on racial discrimination, two identical resumes differing only by name or ID photo are sent to randomly selected firms. A statistical test is then run on the number of interview offers received by each resume to determine if any difference in the number of interview offers received between the resumes holds true in the population.

I am curious whether, if in the event that each resume is sent to a different number of companies (e.g. one is sent to 300 firms and the other 310), results that show a statistically significant difference in the number of interview offers received on one resume compared to the other will be invalidated. My intuition is that there may be a tendency for bias to be introduced as the resume that is assigned to a greater number of firms has a higher chance of being selected. Would appreciate comments on whether this is accurate.

Answer 1

No, this should not be a major issue. We will very carefully take into account the different sample sizes. Allow me to continue your example:

Assume "Resume A" is the the treatment resume and "Resume B" is the control resume, where the treatment has a less advantaged ethnic name (Jamal, Beyonce) and the control resume contains a anglicized name (James, Sophia). One should be careful about gender here as well.

We then compare the mean acceptance rate between the two resumes. Suppose $N_a = 300$ and $N_b = 310$. The accepted resumes are: $Accepted_a = 30$ and $Accepted_b = 62$. For shortcuts, the mean rates of acceptance are $r_a = 30/300 = 0.1$ and $r_b = 62/310 = 0.2$. Notice the mean rate of acceptance normalizes each group, so size is partially accounted for here.

The variances are then:

$\sigma_a^2 = (1-r_a)*r_a = 0.090$

$\sigma_b^2 = (1-r_b)*r_b = 0.160$

And more importantly, the variance of the means of each distribution are then $\sigma_a^2/N_a$ and $\sigma_b^2/N_b$, respectively. We are comparing the means of the two distributions with a simple mean comparison test. Notice this step explicitly uses the sample size to account for expected variation between the two distributions.

So we now compare the two to see if there is a difference, under the null hypotheses there is no difference. (In statistics, you nearly always always assume no difference/no relevance/no effect as the baseline.)

$t=\frac{r_a-r_b}{\sqrt{\sigma_a^2/N_a+\sigma_b^2/N_b}} = 3.572$

The reasoning can be very explicitly laid out as follows:

• The difference between the two samples is 3.572 standard deviations away from 0.
• Under the assumptions we have made above, it seems very unlikely that this would occur by chance if we repeated the process (looking this up on the t-table as a one-tailed test has a p-value of less than 0.001). Bayesians will point out there are some other silent assumptions one may have made, so language is particularly important at this step, but this generally passes the intuition.
• Therefore, I reject the idea that these two samples came from the same distribution. It seems implausible.
• So, I am left to conclude that there is a difference between the two samples. It seems that indeed people are more accepting of the Anglicized names rather than the ethnic names.

Answer 2

They generally measure the probability of getting an interview, not the number of interviews, so that that the number of applications is normalized out. For example, consider Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination (AER 2004 with ~5000 citations).

We study race in the labor market by sending fictitious resumes to help-wanted ads in Boston and Chicago newspapers. To manipulate perceived race, resumes are randomly assigned African-American- or White-sounding names. White names receive 50 percent more callbacks for interviews. Callbacks are also more respon- sive to resume quality for White names than for African-American ones. The racial gap is uniform across occupation, industry, and employer size. We also find little evidence that employers are inferring social class from the names. Differential treatment by race still appears to still be prominent in the U.S. labor market

That paper uses "Likelihood of a Callback" and "Callback Rate" as the primary variables of interest, and these measures use callbacks normalized by applications which means that they are not sensitive to the number of applications (in the levels as least, but the number of applications influences the standard errors).

Answer 3

There are legitimate reasons for wanting to send out more to one group than the other - for example, if you need a sample size of at least 100 to have sufficient power to use certain tests, and you wanted to be able to do some in-group comparisons, you might oversample one or the other.

Example: You have 1000 jobs available to sample. You want to test the following:

• English names vs. Hispanic names
• English names vs. Black names
• English names vs. Black and Hispanic names
• Black names vs. Hispanic names
• Black nonhispanic names vs. Black hispanic names
• Black male names vs. Black female names

And the desired power requires at least n=100. So you might take:

• 300 English (gender irrelevant)
• 250 Black Male inc. 100 Black Hispanic Male
• 250 Black Female inc. 100 Black Hispanic Female
• 200 White Hispanic (gender irrelevant)

That gives us a sufficient power to test quite a few things within groups, while still having the larger out-of-group samples with better power (n=300 vs n=500 for English vs. Black, for example.)

When we do a statistical test, then, the n-sizes of both groups do factor into the test, but they're constructed to consider that. As such, the difference in n sizes is not a reason in and of itself to invalidate a study; it may be for good reasons, or it may simply be a minor irrelevant difference.