There's a formula that shows up everywhere from ZIP files to machine learning to black hole physics, and it looks like this: H = −Σ p log₂ p. Most people meet it, wince, and move on. That's a shame, because underneath the symbols is one of the friendliest ideas in all of math: entropy is just average surprise. Give us two coins and ten minutes, and the formula will feel obvious.

This article is part of a cluster on information and bits. The big picture — Shannon's 1948 insight that information is measurable uncertainty, not meaning — is in the pillar article, what information really is. Entropy is the engine room of that idea, and it directly powers how compression works and why noise limits communication.

Reading the formula out loud

Take it symbol by symbol:

H is the entropy — the number we're computing. It's measured in bits, and it means "the average information per outcome from this random source."

Σ (sigma) just means "add up over all possible outcomes." If your source is a coin, there are two outcomes to sum over. A die, six. The letters of English text, twenty-six and change.

p is the probability of one particular outcome — how often it happens in the long run. A fair coin: p = 0.5 for heads, 0.5 for tails.

log₂ p is the logarithm base 2 of that probability. Since p is at most 1, this number is zero or negative — which is why there's a minus sign out front, purely to flip the final answer positive. No mystery in the minus sign; it's bookkeeping.

Honestly, the formula is friendlier written the other way around:

H = Σ p · log₂(1/p)

Read it as a recipe: for each outcome, compute its surprise, log₂(1/p) — big when p is small, zero when p = 1 — then average the surprises, weighting each by how often it occurs. That's it. Entropy is the expected surprise of the next result.

Why is log₂(1/p) the right measure of surprise? Because it counts yes/no questions. An outcome with probability 1/8 is like one particular leaf on a three-level tree of halvings: it takes three fair yes/no questions to pin down, so learning it happened is worth log₂(8) = 3 bits. Surprise, in this theory, is literally the number of halvings of your uncertainty.

Two coins on the table

Coin one: fair. Heads and tails each have p = 0.5.

Surprise of heads: log₂(1/0.5) = log₂ 2 = 1 bit. Same for tails. Average:

H = 0.5 × 1 + 0.5 × 1 = 1 bit per flip.

Clean and satisfying: every flip of a fair coin delivers exactly one bit, one yes/no question's worth of news.

Coin two: biased, 90% heads, 10% tails.

Surprise of heads: log₂(1/0.9) ≈ 0.152 bits. Barely news at all — you saw it coming. Surprise of tails: log₂(1/0.1) ≈ 3.32 bits. Genuinely surprising — a 1-in-10 event.

Now average, weighted by frequency:

H = 0.9 × 0.152 + 0.1 × 3.32 ≈ 0.137 + 0.332 ≈ 0.47 bits per flip.

Look at what happened. The biased coin's rare outcome is individually MORE surprising than anything the fair coin can do — 3.32 bits versus 1. But it almost never happens, and the outcome that dominates (heads, 90% of the time) is nearly worthless as news. The weighted average lands under half a bit. The fair coin, whose outcomes are individually modest, wins on average.

Push the bias further and entropy keeps falling: a 99/1 coin has about 0.08 bits per flip. A two-headed coin has exactly zero — you know the result before the flip, so the flip tells you nothing. And you can't go the other way: for two outcomes, 1 bit is the ceiling, achieved only at 50/50. Entropy is maximized when you're maximally ignorant.

That's the honest way to say why fair coins carry more entropy: entropy measures uncertainty, and fairness is peak uncertainty. Against a biased coin you have a winning guessing strategy; against a fair coin you have nothing.

Why anyone cares: entropy is the floor

Here's what makes H more than a curiosity. Shannon proved that a source's entropy is the absolute limit on lossless compression: you cannot encode the source's output in fewer than H bits per symbol on average, no matter how clever your scheme, and you can get as close to H as you like.

The biased coin makes this concrete. Naively, recording 1000 flips takes 1000 bits — one per flip. But the 90/10 coin's entropy says the true information content is only about 470 bits. Sure enough, there's a trick: instead of recording each flip, record the flips in blocks, giving short codewords to the common patterns (lots of heads) and long codewords to the rare ones. Done well, you approach 470 bits for the thousand flips. Done for English text — whose entropy is a fraction of the naive 4.7 bits per letter, because letters are highly predictable from context — the same trick becomes ZIP. The machinery of actually building such codes, Huffman trees and dictionary methods, is the subject of how compression works.

Entropy also sets the terms of battle against noise. A channel's capacity — the maximum entropy it can reliably deliver per second — is a hard number, and error-correction codes are how engineers spend some of that capacity buying reliability. This kind of "one formula, ten unexpected consequences" story is exactly the sort of thing microlearning apps like NerdSip love to serve in five-minute sips, and entropy is a rich one.

The other entropy

Now the elephant in the room. Physics has an entropy too — the one in the second law of thermodynamics, the one that always increases, the one associated with disorder and the heat death of the universe. Same word, same-shaped formula. Are they the same thing?

Mathematically, they're close relatives. Boltzmann's entropy is S = k ln W, where W counts the number of microscopic arrangements (positions and speeds of molecules) consistent with what you can measure macroscopically (temperature, pressure), and k is Boltzmann's constant. That's a logarithm of counted possibilities — structurally the equal-probability special case of Shannon's formula, dressed in physical units. The story goes that John von Neumann told Shannon to call his quantity "entropy" partly because of the resemblance, and partly because, as he allegedly quipped, nobody really knows what entropy is anyway.

The connection is more than a joke. You can read thermodynamic entropy as the number of bits you'd need to specify a system's exact microstate given only its macrostate — entropy as missing information. E. T. Jaynes built a serious school of statistical mechanics on exactly that reading, and Landauer's principle ties the two together physically: erasing one bit of information costs a minimum of k T ln 2 in dissipated energy.

But conceptually, keep them in separate pockets. Thermodynamic entropy is a property of physical systems, measured in joules per kelvin, and it explains why heat flows downhill and why perpetual motion machines fail. Shannon entropy is a property of probability distributions, measured in bits, and it explains why your files compress and your video calls survive bad connections. The physics side — where the second law comes from and what "disorder" really means — is covered properly in what is entropy over in the thermodynamics cluster. Bringing the two entropies together rigorously is real, beautiful physics; assuming they're casually interchangeable is a classic way to confuse yourself.

The takeaway

Shannon entropy is average surprise: H = Σ p log₂(1/p), the expected number of yes/no questions the next outcome will answer for you. Predictable sources have low entropy because their outcomes are mostly non-news; a fair coin hits the two-outcome maximum of exactly 1 bit per flip because you can never guess it better than chance. The number isn't just descriptive — it's a law. No lossless code can beat H bits per symbol, and every compressor, channel code, and communication link ever built is playing against that limit. Two coins, one sum, and you've understood the constraint that governs all of digital life.