The Math Behind Neural Networks: Weights, Gradients, and Backpropagation

A neural network can recognize your face, translate a sentence, or predict the next word you're about to type. So where's the intelligence hiding?

Not in some clever rule written by a programmer. It's hiding in a giant pile of plain numbers, weights, that started out random and got nudged, one tiny correction at a time, until they happened to work.

This post is about that nudging. What a weight actually is, how the network figures out which way to nudge it, and why that process has a name you've probably heard without quite trusting: backpropagation.

What is a single neuron, really?

Forget the brain metaphor for a second. A neuron in a neural network does exactly two things, in order.

First, it takes some inputs, multiplies each one by a number, and adds them up. Those multipliers are the weights, and there's one more number added on top called the bias. Second, it squashes that sum through a simple curve so the output stays in a controlled range.

We write the first part as:

And the second part, the squashing, we write as $y = f(z)$, where $f$ is called an activation function. A common choice is the sigmoid curve, which squeezes any number into somewhere between 0 and 1.

Play with $w$ and $b$ above. Notice that $w$ tilts the curve, a bigger weight makes the transition from 0 to 1 sharper, a negative weight flips it. And $b$ just slides the whole curve left or right. That's it. That's the entire vocabulary of a neuron: tilt and slide.

A full network is just thousands of these neurons wired together in layers, each one's output feeding the next layer's inputs. The "intelligence" is entirely in the specific values of every $w$ and $b$ in the whole stack.

So how does the network know which numbers to use?

It doesn't, not at first. Every weight starts out as a small random number. The network's first guess on anything is basically noise.

But we can measure how wrong that guess is. If the network predicts $0.2$ and the correct answer was $1.0$, we can write down a number that captures how bad that miss was, called the loss. A common choice:

Squaring it means small misses barely matter and big misses matter a lot, and it's always positive. Training a neural network means one thing: making this loss number as small as possible, by adjusting every weight in the network.

But here's the immediate problem. A real network might have millions of weights. Which ones do we nudge, and in which direction, and by how much?

The gradient tells you which way is downhill

Here's the idea that makes this tractable. Imagine plotting the loss as a function of just one weight, holding everything else in the network fixed. You'd get a curve, and somewhere on that curve is a valley, the weight value that makes the loss smallest.

You don't need to see the whole curve to know which way to step. You just need the slope under your feet right now. If the slope is positive, the loss goes up as the weight increases, so step left. If the slope is negative, step right.

That slope is the gradient, written $\frac{\partial L}{\partial w}$. And the update rule is beautifully simple:

Here $\eta$ (eta) is the learning rate, how big a step you take. Try it below.

Notice what happens at a small learning rate: slow, steady progress toward the minimum. Now push $\eta$ up toward 1... and the steps start overshooting the valley, bouncing back and forth. Push it past 1 and it diverges, the weight flies off to infinity. The gradient tells you the direction. The learning rate decides whether you get there gracefully or blow past it.

But a network has millions of weights. One slope isn't enough

Fair. And this is where it gets interesting, because most of those weights aren't directly connected to the loss at all. A weight in the very first layer only affects the loss indirectly, through everything downstream of it.

...and the tool for handling exactly this situation is something you already know: the chain rule. If changing a weight changes a hidden neuron's output, and that hidden neuron's output changes the final prediction, and the prediction changes the loss, then:

Each factor is easy to compute on its own. The trick is just multiplying them together in the right order, layer by layer. That multiplication, run backward through the whole network, is what "backpropagation" means. Not a mysterious algorithm. Just the chain rule, applied systematically.

Watching the signal travel backward

The cleanest way to see this is to watch it happen on a tiny network. Below is one input layer, one hidden layer, and one output, wired together with fixed weights.

Hit Step to run the forward pass first. Watch the numbers flow left to right: each input gets multiplied by its edge weight, summed, squashed, and passed to the next layer, until you get a final prediction.

Now hit Step again for the backward pass. The error at the output splits into blame for each incoming edge, that blame becomes a new error signal at the hidden neurons, and that splits into blame for the edges before it. Every edge gets exactly the gradient the chain rule says it should, computed by working backward one layer at a time instead of restarting the whole calculation from scratch for every single weight.

That reuse is the entire point. Without it, training a network with millions of weights would mean redoing the same multiplication millions of times over. Backpropagation computes every gradient in one backward sweep.

Why the activation function matters more than you'd think

There's a catch buried in that chain rule multiplication. Every layer you go back through, you multiply by another derivative of the activation function. If that derivative is small, say $0.1$, then five layers back you're multiplying by roughly $0.1^5 = 0.00001$. The gradient has essentially vanished.

This is called the vanishing gradient problem, and it's why the choice of activation function isn't just cosmetic.

Drag the point along the sigmoid curve toward either edge. Out at $z = 4$ or $z = -4$, the curve is nearly flat, so the slope is tiny, and the "after 5 layers" readout collapses toward zero. Now switch to ReLU. Its slope is either exactly 1 or exactly 0, no matter how far out you go, so a gradient that's flowing at all doesn't shrink as it passes through. That's the real reason ReLU became the default activation in deep networks: it stops the chain rule from quietly erasing the signal.

Putting the whole loop together

Here's the full training loop, now that every piece has a name:

1. Run a forward pass, computing each neuron's weighted sum and activation, layer by layer, until you get a prediction.
2. Measure the loss, how far that prediction was from the target.
3. Run backpropagation, using the chain rule to compute the gradient of the loss with respect to every single weight, working backward from the output.
4. Update every weight a small step against its gradient, scaled by the learning rate.
5. Repeat, thousands or millions of times, on thousands or millions of examples.

Nothing in that loop understands faces or language or anything else. It's the same five steps, indifferent to what the numbers represent, run over and over until the weights happen to encode something useful.

The short version

A neuron multiplies its inputs by weights, adds a bias, and squashes the result through an activation function. Training means adjusting every weight to shrink a loss number, and the only way to know which direction to adjust a weight is its gradient, the slope of the loss with respect to that weight. Backpropagation is just the chain rule, applied layer by layer from the output backward, so every gradient in a massive network can be computed in a single backward pass instead of recomputed from scratch. The activation function's derivative rides along on every one of those multiplications, which is why the wrong choice can make the gradient vanish before it ever reaches the earliest layers.

That's the whole mechanism. Tilt, slide, measure the miss, feel the slope, step downhill, repeat.

All visualizations are interactive React components running entirely in your browser. The network diagram computes real forward and backward passes with fixed weights using plain JavaScript, no machine learning library involved. No libraries beyond React.