Neural Networks and Backpropagation

A handful of borrowed biology and one clever piece of calculus turned out, fifty years later, to be the most consequential algorithm of the era.

For most of the twentieth century, artificial intelligence was about logic — symbols, rules, theorems, proofs. The idea that a machine might think like a brain was not new, but it was treated by most serious researchers as an empty metaphor. What changed is that a small group of people took the metaphor literally enough to build something with it, then found a procedure that could teach the something to be useful. The result is the world you live in now: language models, image generators, voice assistants, cars that drive themselves. The two halves of that story are the neural network and backpropagation.

The neuron — a borrowed metaphor

In 1943, two researchers in Chicago — the neuroscientist Warren McCulloch and the logician Walter Pitts — wrote a paper titled “A Logical Calculus of the Ideas Immanent in Nervous Activity.” They proposed the crudest possible model of a brain cell. Each artificial neuron would receive inputs from other neurons, multiply each input by a weight, sum them, and fire an output if the total exceeded a threshold. That was all.

It was a wild simplification — real neurons are vastly more intricate — but it had two properties that mattered. It was mathematically clean, and it could, in principle, implement any logical function. In 1958 Frank Rosenblatt built a physical machine around the idea, called the Perceptron, and trained it to recognize shapes by adjusting its weights. The press got excited; the New York Times reported that the Navy expected the Perceptron to soon walk, talk, see, write, and reproduce itself.

It did none of those things. In 1969 Marvin Minsky and Seymour Papert proved that a single layer of perceptrons could not even compute the exclusive-or function. The field went into a freeze. The fix was obvious to anyone who looked at it — stack more layers — but nobody knew how to train the stack. With many layers, which weight in which hidden layer should you blame when the network gets an answer wrong? Without an answer to that, deeper networks were just elegant uselessness.

The forward pass — how the network thinks

Strip the biology away and a modern neural network is a long chain of matrix multiplications, with a simple nonlinear function squashed in between each one. Inputs — the pixels of an image, the tokens of a sentence — come in at one end. They get multiplied by a matrix of weights, passed through a nonlinearity, multiplied by the next matrix, passed through the next nonlinearity, and so on. Whatever comes out the other end is the network's answer: a probability that this image is a cat, a guess at the next word.

The architecture is laughably simple. The work is all in the weights — sometimes hundreds of billions of them — that have to be set to the right numbers. With those numbers right, the network can recognize faces, translate Mandarin, fold proteins, beat humans at Go. With them wrong, it outputs gibberish. There is no clever code in between. There is just the matrix arithmetic, and the numbers.

So the entire problem of artificial intelligence becomes a question about the numbers: how do we find them?

Backpropagation — the engine of learning

The procedure that finds them is called backpropagation, and the first time it appeared in a form anyone noticed was 1986, in a short paper in Nature by David Rumelhart, Geoffrey Hinton, and Ronald Williams. (Paul Werbos had derived a version in his 1974 PhD thesis; Seppo Linnainmaa had done the underlying math even earlier. The idea has the usual untidy history.) The trick is high-school calculus, applied with discipline.

Here is the picture. Show the untrained network an input it should know — a picture of a cat — and let it predict. Compare its prediction to the right answer. The difference is the loss: a single number measuring how wrong the network was. The goal is to push every weight in the direction that makes the loss smaller.

The question is how much each weight contributed to the error. Backpropagation answers it by running the chain rule of calculus backward through the network, one layer at a time. The output layer's error tells you which weights in the last layer to blame, and by how much. Those blamed weights then tell you which weights in the previous layer to blame. And so on, all the way back to the inputs. After one backward pass you have a gradient: a tiny nudge for every single weight in the network. Apply the nudges. Show the next example. Repeat for a few trillion examples.

The whole edifice of modern AI sits on a single observation: if your network is built from differentiable pieces, the chain rule will tell you, for free, how to improve every part of it at once.

This is gradient descent, and it is breathtakingly stupid in the way only the right ideas are. There is no model of language, no theory of vision, no careful engineering of features. There is just the rule: nudge every weight a little in the direction that reduces the error. Iterate. Iterate. Iterate. With enough examples and enough weights and enough patience, the network learns whatever the data quietly contains.

Why it matters

For two decades after Rumelhart, Hinton and Williams, almost nobody believed this would scale. Backpropagation worked on toy problems, but real networks of any depth seemed to get stuck or fall apart. The mainstream of AI moved on to other things. Hinton kept at it anyway, mostly in obscurity, mostly in Toronto.

What turned the world over was not a better algorithm but more of the same thing. In 2012, a network called AlexNet — built by Hinton's student Alex Krizhevsky — trained on graphics cards on the ImageNet dataset and crushed the standing image-recognition benchmark by a margin nobody had thought possible. Backpropagation had not changed. The data had grown, and the compute had grown, and that turned out to be all that was missing. Rich Sutton later called this the “bitter lesson”: in the long run, simple methods that scale with computation always beat clever methods that don't.

The same algorithm that trained AlexNet trains everything that came after. Transformers, large language models, AlphaFold, image generators, the model writing this sentence — all of it is, underneath, a stack of differentiable matrix multiplications being nudged into shape by the chain rule running backwards. Almost the entire visible surface of modern AI is one idea from 1986 applied at a scale its inventors did not imagine.

It is worth pausing on how strange this is. A crude metaphor for a neuron, plus a calculus trick a high schooler can follow, somehow contain enough structure to learn the patterns of human language from a slab of internet text. Nobody understands quite why it works as well as it does. We know how to do it. We do not yet know what it means that we can.