My question is regarding in the difference in difference model, I am trying to see the impact of serial correlation within subjects on the treatment effect inference.

In my simulation, I have two periods (pre & post) and two cohorts (control & treatment). In the diff-in-diff model, I include three fixed effects, which are 'period', 'cohort' and 'treated', all of these three fixed effect are binary variable, for each subject in the simulation, their pre & post values are positively correlated, and no treatment effect & pre-post change is used in the simulation.

I think this setup would result in a positively correlated residuals within each subject, and no correlation across subject, and from literature, I would expect to see an inflated p-value(greater than 0.05 false positive under the 0.05 threshold). However, I am not sure why I saw a deflated p-value instead ? a much smaller p-value than 0.05.

Thanks for your help !

pvalues = []
effect = 0
pre_mean = 10
post_mean = pre_mean + effect
var = 10
pre_post_cov = 5

# number of subject/samples do we have in total, each subjects has pre & post period observation
N = 5000

for i in range(1000):

    samples = np.random.multivariate_normal(
        cov=[[var, pre_post_cov], [pre_post_cov, var]], 

    pre = samples[:,0]
    post = samples[:,1]
    subjects = np.concatenate([np.arange(N) , np.arange(N)])
    periods = [0]*int(N/2) + [0]*int(N/2) + [1]*int(N/2) + [1]*int(N/2)
    periods = np.array(periods)
    # Assume half are control, the rest half are test.
    cohorts = [0]*int(N/2) + [1]*int(N/2) + [0]*int(N/2) + [1]*int(N/2)
    cohorts = np.array(cohorts)
    cohorts_period = cohorts * periods
    Y = np.concatenate([pre, post])
    X = np.array([cohorts, periods, cohorts_period]).T
    X = sm.add_constant(X)
    model = sm.OLS(Y, X)
    results = model.fit()


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.