What Is Entropy in Information Theory?

Can you measure surprise? Not the feeling, the actual number. If someone tells you it rained today in Seattle, you shrug. If they tell you it rained today in the Sahara, you sit up. Same sentence structure, same two words "it rained." Wildly different amount of information.

Information theory says yes, you can measure it. In bits. And the tool that does it is called entropy. It sounds intimidating because it's borrowed from physics, but the information-theory version is a much simpler idea once you see it from the right angle.

Rare events carry more information than common ones

Let's start with the Seattle and Sahara example, because it already contains the whole idea. Rain in Seattle happens all the time. Learning that it rained barely narrows down anything about the world, you basically expected it. Rain in the Sahara is rare. Learning it happened rules out a huge number of "normal" days you were expecting instead.

That's the core intuition: information is about how much a message narrows down your uncertainty. A message that confirms what you already expected gives you almost nothing. A message that overturns your expectations gives you a lot.

Now here's a game that makes this precise: Twenty Questions. Someone picks a secret, you ask yes or no questions to figure out what it is. If the secret is drawn from a small, predictable set of options, you can nail it in a couple of questions. If it could be almost anything, you need a lot more questions on average.

Entropy is going to turn out to be exactly that: the average number of yes or no questions you need, if you play optimally. Keep that game in your head. We'll come back to it.

Turning "surprising" into an actual number

Before we can average anything, we need a way to score a single outcome's surprise. Let's think about what a good surprise-score should do.

If an outcome is certain, probability $p = 1$, it should score zero surprise. You already knew it was coming. If an outcome is rare, small $p$, it should score high. And here's a property that matters a lot: if two independent events both happen, like flipping heads twice in a row, the total surprise should just add up. Two independent surprises stack.

Probabilities don't add when you combine independent events, they multiply: $p(\text{heads twice}) = p \cdot p$. But surprise needs to add. What turns multiplication into addition? A logarithm. So we define:

$\text{Surprise}(p) = -\log_2 p$

The minus sign is just there because $\log_2 p$ is negative for any $p$ under 1, and we want a positive number of bits. We use base 2 specifically because then the units come out in bits, the same bits a computer uses. An outcome with $p = 0.5$ scores exactly 1 bit of surprise. An outcome with $p = 0.25$ scores 2 bits. Each time you halve the probability, you add exactly one more bit.

Try dragging the point below along the curve and watch how fast surprise climbs as probability shrinks.

Notice the "1 in N" readout next to the curve. That's not a coincidence, it's the whole point of the formula: an outcome with probability $1/8$ needs 3 bits, because $2^3 = 8$. Surprise in bits literally counts how many times you'd have to cut the space of possibilities in half to land on this one outcome. Which, notice, is exactly what a yes or no question does.

A loaded coin is more predictable, and less uncertain

Now let's average this surprise score over every possible outcome, weighted by how likely each one is. Start with the simplest case: a coin that lands heads with probability $p$ and tails with probability $1-p$.

$H(p) = -p\log_2 p - (1-p)\log_2(1-p)$

Don't worry about memorizing that. Just read it as: "surprise of heads, weighted by how often heads happens, plus surprise of tails, weighted by how often tails happens." That weighted sum is the entropy of the coin, in bits.

Play with the bias below. Watch what happens as the coin drifts from fair toward one-sided.

Here's the thing worth sitting with: entropy peaks exactly at $p = 0.5$, at 1 full bit. A fair coin is the single most unpredictable coin you can have. Tilt it either direction, toward mostly-heads or mostly-tails, and entropy falls. Push it all the way to $p = 0$ or $p = 1$ and entropy hits zero: a coin that always lands the same way tells you nothing when you flip it, because you already knew the answer.

That matches the Twenty Questions intuition perfectly. A fair coin costs you exactly 1 question to resolve, no shortcuts available. A heavily biased coin, you could often just guess "heads" and be right most of the time, needing a real question far less than once per flip on average.

Entropy is just weighted-average surprise

The coin was a warm-up. Real distributions usually have more than two outcomes, so let's generalize. For any set of outcomes with probabilities $p_1, p_2, \ldots, p_n$, entropy is the probability-weighted average of every outcome's surprise:

$H = -\sum_i p_i \log_2 p_i$

That's it. That's the whole formula. It's the same weighted-average idea you'd use to compute an average test score, except instead of averaging scores you're averaging "bits of surprise," and instead of weighting by how many students got that score, you're weighting by how likely that outcome is.

Try it yourself on a loaded die. Drag any bar and watch the rest rebalance so the probabilities still add up to 1.

A fair six-sided die gives you the maximum possible entropy for six outcomes, $\log_2 6 \approx 2.585$ bits. Load the die so one face comes up 85% of the time, and entropy collapses, because most rolls now tell you almost nothing. You already knew what was probably coming.

Guessing games and entropy are the same idea in disguise

Let's close the loop back to Twenty Questions. Suppose there are four possible outcomes, A, B, C, and D, and you get to design your questions in advance. If you're smart about it, you don't split the outcomes 50/50 blindly. You ask about the most likely outcome first.

"Is it A?" If A is very likely, you'll often get a yes and finish in one question. If not, you move to "Is it B?", then "Is it C?", spending extra questions only on the rare cases. This is exactly how good compression and good coding schemes work: spend fewer bits on common things, more bits on rare things.

Watch the "extra questions wasted" readout as you drag the skew slider. This particular tree strategy, always asking about the single most likely remaining outcome, isn't always perfectly optimal, but it gets close. And notice, at the extremes, when one outcome dominates almost completely, the tree's average number of questions converges right down toward the true entropy.

That convergence isn't a coincidence either. It's a real theorem, called the source coding theorem: entropy is the theoretical floor for the average number of yes or no questions (or bits) any strategy needs to identify an outcome. No cleverer coding scheme can beat it. Entropy isn't just a summary statistic, it's a hard limit, the same way the speed of light is a hard limit on how fast you can send a signal.

So, in plain English

Entropy measures how much genuine uncertainty sits inside a probability distribution, in bits. Each possible outcome carries a "surprise" score of $-\log_2 p$, bigger for rarer outcomes. Entropy is just the average of that surprise, weighted by how often each outcome actually happens. A predictable source, a biased coin, a loaded die, a rainy day in Seattle, has low entropy: you already know roughly what's coming, so there's not much left to tell you. A source where anything could happen with equal chance has maximum entropy, and that number also happens to be exactly how many yes or no questions you'd need, on average, to nail down the outcome by playing optimally.