Entropy as information

There is a very concrete way to put a number on our ignorance about the microstate: count how many yes/no questions are needed, on average, to identify it. This idea links the entropy of thermodynamics to the measurement of information.

Imagine a bag with 3 balls: 2 blue and 1 red, all identical to the touch. How many microstates does this macrostate have? They are the ways of arranging the 3 balls in 3 distinguishable positions:

$\Omega = \frac{3!}{2!\cdot 1!} = 3$

To find out which ball is in which position takes, on average, $\log_2 3 \approx 1{,}58$ yes/no questions (logarithms base 2 count exactly the number of binary questions).

With 6 balls (4 red, 2 blue) the number of microstates grows:

$\Omega = \frac{6!}{4!\cdot 2!} = 15 \qquad \log_2 15 \approx 3{,}9 \text{ questions}$

Entropy as a number of questions

The more microstates there are, the more questions are needed to identify the true one: in this sense, entropy is the amount of information we lack about the system.

This idea — entropy as the number of questions required — is the same one we use today to measure the security of a password: a 4-digit password has “information entropy” $\log_2 10^4 \approx 13{,}3$ bits. It is striking that nineteenth-century thermodynamics and twentieth-century information theory, born for completely different purposes, are so deeply connected: in both, entropy measures how much we do not know about a system with many possible configurations.

Links

Topics: Entropy and the second law Concepts: Entropy Skills: Micro-macro interpretation

Related exercises: Worked exercise — the coffee cools down · Hot shower: ΔS of the universe · Heat from hot to cold: proof