Options:

1. The skellam distribution is the distribution of the difference in Poisson variables. You should be able construct a distribution and then get the middle 95% by evaluating inverse cdf (0.025) and icdf(0.975)

https://en.m.wikipedia.org/wiki/Skellam_distribution

2) assume normality, which is pretty reasonable given the sample size. Mean is u1 - u2, and variance is u1 + u2. This is the simplest, but doesn't account for the block correlation.

3) run a Clustered bootstrap: similar to a simulation except you redraw from the sample and draw the clusters together. Then take quantiles of the difference between the two outcomes. This is the most accurate.

See cluster data block bootstrap on wikipedia. It sounds tricky, but it's literally just a couple lines of Python.

Cluster data describes data where many observations per unit are observed. This could be observing many firms in many states, or observing students in many classes https://en.m.wikipedia.org/wiki/Bootstrapping_(statistics)

More on reddit.com

You just need to use the Finite Population Correction Factor p̂ ± Z(α/2) * √(p̂(1-p̂) / n) * √((N-n)/(N-1) More on reddit.com

For users of old reddit: The table and code in the question might not work if you use old reddit (at least for some setups). This might save you some effort: Group | Converted | Did not convert | Control | 30 | 387 | Test | 59 | 465 code under 1: ctrl <- 30/(30 + 387) test <- 59/(59 + 465) sqrt(ctrl*(1-ctrl)/(30 + 387)) # 0.01265354 sqrt(test*(1-test)/(59 + 465)) # 0.01380879 and under 2: calculate the confidence interval of each proportion ctrl +/- 2 * 0.01265354 <-- 2 standard devs gives 95% conf interval test +/- 2 * 0.01380879 On the CI for the difference in proportion: The use of two confidence intervals in Step 3 is not correct. You can form an approximate confidence interval for the difference in proportion by adding the squares of the standard errors of each proportion and taking the square root (giving the s.e. of the difference), and using a z interval based off that. There are other approaches but that should work fine. Since you're using R, you can skip that and just use prop.test which by default gives the same chi-squared value as chisq.test (i.e. it also uses Yates' continuity correction, though doesn't say so in the output), but then it also gives an interval for the difference in proportion: conv<-matrix(c(30,59,387,465),nr=2) rownames(conv)<-c("Control","Test") colnames(conv)<-c("Converted","Did not convert") conv prop.test(conv) The output for the last two lines should look like this: > conv Converted Did not convert Control 30 387 Test 59 465 and > prop.test(conv) 2-sample test for equality of proportions with continuity correction data: conv X-squared = 4.0192, df = 1, p-value = 0.04498 alternative hypothesis: two.sided 95 percent confidence interval: -0.079515385 -0.001790562 sample estimates: prop 1 prop 2 0.07194245 0.11259542 so with those settings, the 95% CI does not overlap 0. You can turn off the continuity correction in the usual way (i.e. just as with chisq.test). More on reddit.com

Initially, I calculated the sample size required for 95% confidence interval, with a margin of error of 5%. I get 384 files to review using a standard online calculator. What do you mean by this? You can always calculate a 95% confidence interval as long as you have more observations than parameters in your model. Did you do some kind of power calculation? More on reddit.com