Problem — How many repetitions to confirm a law

Problem

Estimate, using Fermi’s method, how many independent measurements are needed to confirm a new law at $99\%$ confidence. Consider first the simple case (binary test, heads/tails-type), then the case of continuous estimates with error bars.

Solution

Binary case (yes/no). Suppose a prediction of the law has a $50\%$ probability of turning out correct by pure chance (like guessing heads or tails). If we repeat $n$ independent trials and the law gets it right every time, the probability that this happens by chance is $P = \left(\frac{1}{2}\right)^{n}$ We want to go below $1\%$ so that a chance coincidence becomes implausible: $\left(\frac{1}{2}\right)^{n} < 0{,}01$ Taking the logarithm: $n > \frac{\log(0{,}01)}{\log(0{,}5)} = \frac{-2}{-0{,}301} \approx 6{,}6$ So about 7 independent confirmations suffice to reach $99\%$ confidence in the binary case.

Continuous case (measurements with error bars). When the prediction is a continuous numerical value compared with a measurement affected by uncertainty $\sigma$, a “yes/no” is not enough: the confidence interval on the mean must be narrow enough to distinguish the predicted effect from noise. Since the uncertainty on the mean decreases as $\sigma/\sqrt{N}$, tightening the intervals to the $99\%$ level and resolving effects comparable to $\sigma$ typically requires a few tens of independent measurements (the precise number depends on $\sigma$ and on the size of the effect).

$\ev{n \approx 7 \ (\text{binary}) \;;\quad n \sim \text{tens} \ (\text{continuous})}$

Links

Topics: Metodo sperimentale Concepts: Cifre significative e incertezza Skills: Stime di Fermi