Occam's Razor

A medieval friar's warning against needless entities became, seven centuries later, a theorem about learning from data.

Almost every intelligent person you meet endorses some version of it. Politicians invoke it, detectives use it, physicists live by it. It has a lovely mystique — a razor, sharp and small, that cuts through nonsense. And yet if you press people on what it actually says, or why it should be true, the answers dissolve. Simpler explanations are better. Why? Because they are. It is very nearly a folk theorem: everyone knows it, no one can prove it.

The strange thing is that a proof, or something very close to one, does exist. It just took an unusually long detour through logic, computation, and probability theory to arrive.

What Ockham actually said

William of Ockham was a Franciscan friar working in the early 1300s, entangled in the vicious philosophical disputes of his time about universals, essences, and the metaphysics of God. He never wrote the sentence most often attributed to him — entia non sunt multiplicanda praeter necessitatem, “entities are not to be multiplied beyond necessity.” That formulation was tidied up centuries after his death. What he actually wrote, repeatedly, was closer to: it is futile to do with more what can be done with fewer.

The point was theological before it was scientific. If two explanations of the world both fit the evidence, we have no warrant to posit the extra ghosts, essences, or angels that one of them requires. Faith might; reason cannot. His razor was epistemic humility given a sharp edge: a rule for when to refuse to believe in something.

It is important that this is a rule about explanations, not about the world. Ockham did not claim reality was simple. He claimed that our theories should not add complexity unforced. The world is allowed to be as baroque as it likes; we are just not allowed to invent extra machinery on its behalf.

Why fewer moving parts win

The most compelling everyday defence of the razor is not philosophical but statistical. Suppose you plot some measurements — the yield of a crop against rainfall, say — and try to draw a curve through the points. A straight line will miss most of them by a little. A wiggly polynomial with fifteen coefficients can be made to pass exactly through every point. Which curve do you trust to predict next year?

A complicated model, with enough tunable parameters, can fit any dataset perfectly — including its noise. But noise is by definition the part that will not repeat. A model that fits the noise fits things that were never there to begin with. When you deploy it on new data, it fails. The simpler model, by refusing to bend to every wiggle, leaves the noise where it belongs: outside the theory. This is called overfitting, and it is the reason machine learning practitioners spend enormous effort on regularisation, cross-validation, and information criteria — all of which are, at heart, Occam's razor with equations.

Bayes, Solomonoff, and the mathematics of simplicity

The first serious formalisation is Bayesian. In Bayes' theorem, our belief in a hypothesis is proportional to two things: how well it explains the data, and how much prior weight we gave it. The clever move — sometimes called the automatic Occam's razor of Bayesian inference — is that a hypothesis with more free parameters spreads its predictions over a wider range of possible observations. So when the actual data comes in, a complicated hypothesis assigns only a thin sliver of probability to it, while a simple hypothesis, forced to make a sharper prediction, either wins big or loses big. Averaged across the sharp predictions and the vague ones, simplicity is rewarded automatically. No extra rule needed.

But this only tells us which of two given hypotheses to prefer. It does not tell us how to assign a prior in the first place. Enter Ray Solomonoff, in 1960. He proposed a shocking answer: weight every hypothesis by how short a computer program is needed to describe it. Programs that are short get high prior probability. Programs that are long get exponentially small prior probability. This is Occam's razor turned into arithmetic.

Solomonoff proved that an ideal agent using this prior would, given enough data, converge to the truth about any computable process in the universe faster than any other conceivable predictor. This is not folklore. It is a theorem. The catch, and it is a serious one, is that computing the exact prior is impossible — it depends on Kolmogorov complexity, which is uncomputable. But every practical approximation of it, from MDL to compression-based prediction to modern neural network priors, inherits its spirit: shorter descriptions win.

Simplicity is not an aesthetic preference. It is the shape of any theory of learning that hopes to generalise beyond the data it has seen.

The razor cuts both ways

None of this makes the razor infallible. It is a prior, not a proof. There are domains where the true story really is complicated — particle physics, biology, weather — and a stubborn insistence on simplicity would leave you with theories that are elegant and wrong. Copernicus's original heliocentric model, all circles and no ellipses, was simpler than Ptolemy's and no more accurate; Kepler had to add complexity to get it right. Einstein warned that a theory should be as simple as possible — and no simpler.

What Ockham's razor really does is set the burden of proof. You are free to invoke a soul, an ether, an epicycle, a hidden variable. But the extra machinery must earn its keep in fresh predictions, not just help you fit the data you already have. If it does no work, out it goes. Seven hundred years later, that is still the deal: keep what you need, cut what you can. The blade has changed hands — from Franciscan friar to Bayesian statistician to compression algorithm — but it is the same blade, and it is still very sharp.