Problem — Small sample (coin)

Problem

A student tosses a coin $10$ times, gets $8$ heads, and concludes “it’s rigged”. Why is this a weak conclusion? Reason on the binomial distribution: the probability of $\ge 8$ heads out of $10$ with a fair coin is about $5{,}5\%$.

Solution

The probability at play. With a perfectly fair coin (probability $1/2$ per face), the probability of getting $8$, $9$ or $10$ heads out of $10$ tosses is, summing the three cases of the binomial distribution, about $P(\ge 8 \text{ heads}) \approx 5{,}5\%$ This means the observed event happens on average about once every 18 trials of ten tosses: it is rare, but far from exceptional. It is not at all proof of rigging.

The small-sample problem. A sample of only $10$ trials is subject to strong statistical fluctuations: the results can deviate considerably from the expected value ($5$ heads) purely by chance. With so little data one cannot distinguish a genuinely rigged coin from a random fluctuation of a fair one. The student’s conclusion confuses an ordinary fluctuation with a real effect.

How many trials are needed. The relative size of statistical fluctuations decreases as $1/\sqrt{N}$ as the number $N$ of trials grows. To reduce the noise to a level that allows an imbalance to be detected, many more trials are needed: of the order of $\sim 100$ tosses or more. Only then would a persistent excess of heads become statistically significant.

$\ev{P(\ge 8) \approx 5{,}5\% \Rightarrow \text{ordinary fluctuation; need} \sim 100 \text{ tosses to conclude}}$

Links

Topics: Metodo sperimentale Concepts: Cifre significative e incertezza