In traditional field experiments on racial discrimination, two identical resumes differing only by name or ID photo are sent to randomly selected firms. A statistical test is then run on the number of interview offers received by each resume to determine if any difference in the number of interview offers received between the resumes holds true in the population.

I am curious whether, if in the event that each resume is sent to a different number of companies (e.g. one is sent to 300 firms and the other 310), results that show a statistically significant difference in the number of interview offers received on one resume compared to the other will be invalidated. My intuition is that there may be a tendency for bias to be introduced as the resume that is assigned to a greater number of firms has a higher chance of being selected. Would appreciate comments on whether this is accurate.


3 Answers 3


No, this should not be a major issue. We will very carefully take into account the different sample sizes. Allow me to continue your example:

Assume "Resume A" is the the treatment resume and "Resume B" is the control resume, where the treatment has a less advantaged ethnic name (Jamal, Beyonce) and the control resume contains a anglicized name (James, Sophia). One should be careful about gender here as well.

We then compare the mean acceptance rate between the two resumes. Suppose $N_a = 300$ and $N_b = 310$. The accepted resumes are: $Accepted_a = 30$ and $Accepted_b = 62$. For shortcuts, the mean rates of acceptance are $r_a = 30/300 = 0.1$ and $r_b = 62/310 = 0.2$. Notice the mean rate of acceptance normalizes each group, so size is partially accounted for here.

The variances are then:

$\sigma_a^2 = (1-r_a)*r_a = 0.090$

$\sigma_b^2 = (1-r_b)*r_b = 0.160$

And more importantly, the variance of the means of each distribution are then $\sigma_a^2/N_a$ and $\sigma_b^2/N_b$, respectively. We are comparing the means of the two distributions with a simple mean comparison test. Notice this step explicitly uses the sample size to account for expected variation between the two distributions.

So we now compare the two to see if there is a difference, under the null hypotheses there is no difference. (In statistics, you nearly always always assume no difference/no relevance/no effect as the baseline.)

$t=\frac{r_a-r_b}{\sqrt{\sigma_a^2/N_a+\sigma_b^2/N_b}} = 3.572$

The reasoning can be very explicitly laid out as follows:

  • The difference between the two samples is 3.572 standard deviations away from 0.
  • Under the assumptions we have made above, it seems very unlikely that this would occur by chance if we repeated the process (looking this up on the t-table as a one-tailed test has a p-value of less than 0.001). Bayesians will point out there are some other silent assumptions one may have made, so language is particularly important at this step, but this generally passes the intuition.
  • Therefore, I reject the idea that these two samples came from the same distribution. It seems implausible.
  • So, I am left to conclude that there is a difference between the two samples. It seems that indeed people are more accepting of the Anglicized names rather than the ethnic names.
  • $\begingroup$ Thanks for the really great explanation. Assuming that we work instead with numbers of acceptances rather than normalised rates (as much as it is a poor approach), sample sizes would pose an issue then? Here, I presume that a t-test would be used? $\endgroup$ Commented Oct 2, 2020 at 3:44
  • $\begingroup$ @jo_1 How do you "...work...with numbers of acceptances"? What would be the statistic? The population parameter of interest is the probability of acceptance, conditional on control/treatment. Taking the sample rate is not some "fix" that's supposed to be remedy for small sample size. It's just estimating the population moment by sample moment. (If sample size is a problem and one would like to use the same model, only fix is to get a larger sample.) $\endgroup$
    – Michael
    Commented Oct 2, 2020 at 8:04
  • $\begingroup$ Point taken. Just realized that we would simply not be able to come up with a statistic based on numbers of acceptances. $\endgroup$ Commented Oct 2, 2020 at 10:14
  • 2
    $\begingroup$ Note though: when the sample sizes in the treatment arms get very different, total uncertainty in the difference becomes dominated by the size of the smaller treatment arm. $\endgroup$
    – cbeleites
    Commented Oct 2, 2020 at 14:39
  • 1
    $\begingroup$ @cbeleitesunhappywithSX relatedly, equal sized groups are most often chosen because it gives the highest efficiency. In other words if you were doing a clinical trial on a covid vaccine vs placebo and had n people, the most optimal experiment would be to assign each group n/2 people. there are other considerations though, as Joe's answer points out. $\endgroup$
    – eps
    Commented Oct 2, 2020 at 19:59

They generally measure the probability of getting an interview, not the number of interviews, so that that the number of applications is normalized out. For example, consider Are Emily and Greg More Employable Than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination (AER 2004 with ~5000 citations).

We study race in the labor market by sending fictitious resumes to help-wanted ads in Boston and Chicago newspapers. To manipulate perceived race, resumes are randomly assigned African-American- or White-sounding names. White names receive 50 percent more callbacks for interviews. Callbacks are also more respon- sive to resume quality for White names than for African-American ones. The racial gap is uniform across occupation, industry, and employer size. We also find little evidence that employers are inferring social class from the names. Differential treatment by race still appears to still be prominent in the U.S. labor market

That paper uses "Likelihood of a Callback" and "Callback Rate" as the primary variables of interest, and these measures use callbacks normalized by applications which means that they are not sensitive to the number of applications (in the levels as least, but the number of applications influences the standard errors).

  • $\begingroup$ Thanks for the reply. This means that an experiment that uses raw numbers as its variable of interest is necessarily sensitive to the number of applications? $\endgroup$ Commented Oct 1, 2020 at 15:08
  • $\begingroup$ Yes, unless you are careful to balance the sizes of the groups and sub-populations. Normalization is generally going to be the superior and more flexible approach. $\endgroup$
    – BKay
    Commented Oct 1, 2020 at 15:20

There are legitimate reasons for wanting to send out more to one group than the other - for example, if you need a sample size of at least 100 to have sufficient power to use certain tests, and you wanted to be able to do some in-group comparisons, you might oversample one or the other.

Example: You have 1000 jobs available to sample. You want to test the following:

  • English names vs. Hispanic names
  • English names vs. Black names
  • English names vs. Black and Hispanic names
  • Black names vs. Hispanic names
  • Black nonhispanic names vs. Black hispanic names
  • Black male names vs. Black female names

And the desired power requires at least n=100. So you might take:

  • 300 English (gender irrelevant)
  • 250 Black Male inc. 100 Black Hispanic Male
  • 250 Black Female inc. 100 Black Hispanic Female
  • 200 White Hispanic (gender irrelevant)

That gives us a sufficient power to test quite a few things within groups, while still having the larger out-of-group samples with better power (n=300 vs n=500 for English vs. Black, for example.)

When we do a statistical test, then, the n-sizes of both groups do factor into the test, but they're constructed to consider that. As such, the difference in n sizes is not a reason in and of itself to invalidate a study; it may be for good reasons, or it may simply be a minor irrelevant difference.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.