Randomization Distribution — Definition, Formula & Examples
A randomization distribution is the collection of statistic values you get by repeatedly shuffling your data as if the null hypothesis were true. It shows which results chance alone tends to produce, so you can judge whether your actual result is unusual.
The randomization distribution of a statistic is its distribution over all possible re-randomizations of the observed data that are consistent with the null hypothesis, such as random reassignments of treatment labels to the observed responses. In practice it is approximated by a large number of simulated re-randomizations. It is centered at the null-hypothesized value of the parameter, often 0 for a difference.
Key Formula
Where:
- = Total number of simulated re-randomizations (for example, 1000)
- = Estimated probability of a result at least as extreme as the observed one, assuming the null hypothesis is true
Worked Example
Problem: Ten volunteers are randomly assigned, 5 to caffeine and 5 to a placebo, then count finger taps in 10 seconds. Caffeine: 15, 17, 18, 19, 21. Placebo: 12, 13, 14, 15, 16. Test whether caffeine increases the mean tap count, using a randomization distribution with 1000 shuffles and significance level 0.05.
Step 1: Compute the observed statistic: the caffeine mean minus the placebo mean.
Step 2: Build the randomization distribution. Assume the null hypothesis, that caffeine has no effect, so each person's tap count would have been the same in either group. Pool all 10 values, randomly split them into two groups of 5, and record the difference in means. Repeat 1000 times. The resulting distribution is centered near 0.
Step 3: Suppose 12 of the 1000 shuffled differences are 4 or greater. Use that count to estimate the one-sided p-value.
Answer: The p-value is about 0.012, which is less than 0.05. A difference of 4 or more rarely happens by chance alone, so there is convincing evidence that caffeine increases the mean tap count.
Why It Matters
AP Statistics free-response questions often show a dotplot of simulated differences and ask you to estimate a p-value from it. The same shuffling method, called a permutation test, is how data scientists judge A/B tests when the conditions for a normal model are not met.
Common Mistakes
Mistake: Expecting the randomization distribution to be centered at the observed statistic, such as 4 in the example.
Correction: The shuffles are done assuming the null hypothesis is true, so the distribution centers at the null value, usually 0. A bootstrap distribution, used for confidence intervals, is the one centered near the observed statistic.