Neural Networks and Backpropagation: How a Network Learns

This series started with describing data and worked through regression, classification, resampling, trees and unsupervised learning. It ends with the method behind most of today’s machine learning: the neural network. A neural network is, at heart, a stack of the linear models from the earlier posts with a non-linear function between them, and it is trained the way logistic regression is trained: by following the gradient of a loss downhill. What makes large networks trainable at all is how that gradient is computed: backpropagation.

This post builds networks from a single neuron, derives backpropagation, solves a small example by hand with every number checked against finite differences, and measures the choices that decide whether training works at all: the activation, the learning rate, the batch size and the initial weights. The Computer Vision series continues from here to convolutional networks.

---

1. From One Neuron to a Network

A neuron computes a weighted sum of its inputs plus a bias and passes it through an activation function $g$:

$$h_{\theta }(x)=g(w^{\top }x+b),\theta =(w,b).$$

The order matters: first the linear combination, then the activation. The first artificial neuron, Rosenblatt’s perceptron (1958), used a step function: output 1 if $w^{\top }x+b\ge 0$, else 0. With the sigmoid $g(z)=1/(1+e^{-z})$ the neuron is exactly logistic regression; with the rectified linear unit $\text{ReLU}(z)=\max (z,0)$ it is a linear model that cannot go below zero, and with $\tanh$ it is a rescaled sigmoid between −1 and 1.

A single neuron draws one straight boundary. A network feeds the outputs of several neurons (a hidden layer) into further neurons. With $m$ hidden units and input $x\in {\mathbb{R}}^d$:

$$a=g(W^{(1)}x+b^{(1)})\in {\mathbb{R}}^m,h_{\theta }(x)=w^{(2)\top }a+b^{(2)},$$

with $W^{(1)}\in {\mathbb{R}}^{m\times d}$. Computing the output layer by layer, left to right, is the forward pass. Stacking more hidden layers gives a deep network. The final layer usually has no ReLU when the output can be negative; for classification it is a sigmoid (two classes) or a softmax (several).

The activation is what makes it work. Without it, every layer is linear and so is their composition: $W^{(2)}(W^{(1)}x)=(W^{(2)}{W}^{(1)})x$. On a two-class problem shaped like two interlocking half-moons:

Model (two moons, 4,000 test points)	Test accuracy
Logistic regression	0.8628
16 hidden units, no activation	0.8628 (the same model)
16 hidden units, ReLU	0.9895
16 hidden units, tanh	0.9908

With a non-linear activation and enough hidden units, even a single hidden layer can approximate any continuous function on a bounded region as closely as desired (the universal approximation theorem; Cybenko, 1989; Hornik, 1991). The theorem says such weights exist; it says nothing about how many units are needed, how much data, or whether training will find them.

---

2. Loss and Gradient Descent

Training means choosing $\theta$ to minimise a loss averaged over the training data, such as the squared error for regression,

$$J(\theta )=\frac{1}{n}\sum _{i=1}^n{J}^{(i)}(\theta ),J^{(i)}(\theta )=\frac{1}{2}(h_{\theta }(x^{(i)})-y^{(i)}{)}^2,$$

or the cross-entropy of Classification for classes. (The factor ½ only simplifies the derivative: multiplying a function by a positive constant does not move its minima.)

There is no closed-form solution, so the loss is minimised iteratively by gradient descent: compute the gradient ${\nabla }_{\theta }J$, the direction of steepest ascent, and take a step against it,

$$\theta \leftarrow \theta -\eta {\nabla }_{\theta }J(\theta ),$$

with learning rate $\eta >0$. The loss of a network is not convex, so gradient descent finds a local minimum (or a flat region), which in practice is usually good enough.

The learning rate

The same network on the moons, full-batch gradient descent, 2,000 steps from the same start:

Learning rate	Training loss after 2,000 steps	Test accuracy
0.001	0.394	0.830
0.1	0.082	0.973
1	0.008	0.990
10	0.189	0.883
100	1.644	0.494 (chance)

Too small and training crawls; too large and every step overshoots, until the loss grows instead of shrinking.

Stochastic and mini-batch gradient descent

Computing the gradient over all $n$ examples for every step is expensive when $n$ is large. Stochastic gradient descent (SGD) uses one random example per step, and mini-batch SGD a random batch of $B$ examples. Their gradient is a noisy but unbiased estimate of the full one, and its noise shrinks as $1/B$ (learning rate 0.5, and 0.05 for B = 1, which is unstable at 0.5):

Batch size B	Mean squared error of the gradient estimate	Updates in 30 passes over 400 examples · test accuracy
1	2.368	12,000 · 0.990
8	0.299
32	0.073	375 · 0.971
128	0.019
400 (full batch)	0	30 · 0.863

Two reasons favour mini-batches. Per pass over the data, smaller batches make many more (noisier) updates, so they usually get further for the same computation; and a batch of $B$ examples is computed in one matrix operation, which on modern hardware costs far less than $B$ separate ones. Batches of 32 to a few hundred balance the two.

---

3. Backpropagation

Gradient descent needs ${\nabla }_{\theta }J$: the derivative of the loss with respect to every weight. A network can have millions. Approximating each derivative by nudging one weight and re-running the network (finite differences) needs two forward passes per weight: for a small network with 784 inputs, 100 hidden units and 10 outputs (79,510 parameters), that is 159,020 forward passes per gradient. Backpropagation computes all of them at the cost of about one forward and one backward pass, 1.8 forward passes in a measurement on that network.

It is the chain rule, applied in a particular order. If the loss depends on a parameter through intermediate quantities $g_1,\dots ,g_k$,

$$\frac{\partial J}{\partial {\theta }_i}=\sum _{j=1}^k\frac{\partial J}{\partial g_j}\frac{\partial g_j}{\partial {\theta }_i}.$$

Backpropagation starts at the loss and moves backward through the network, one layer at a time. For each unit it computes an error signal $\delta =\partial J/\partial z$, the derivative of the loss with respect to that unit’s weighted sum, from the error signals of the layer above:

$${\delta }^{(\ell )}=\left(W^{(\ell +1)\top }{\delta }^{(\ell +1)}\right)\odot g^{\prime }\left(z^{(\ell )}\right),\frac{\partial J}{\partial W_{ij}^{(\ell )}}={\delta }_i^{(\ell )}{a}_j^{(\ell -1)}.$$

Every weight’s gradient is the error signal of the unit it feeds into, times the activation of the unit it comes from, both of which are already known from the forward pass.

Two clarifications of common descriptions:

• Backpropagation computes gradients; it does not minimise anything. Minimising is gradient descent’s job (or another optimiser’s). The two are often run together, but are separate: the same backpropagated gradient can feed plain SGD, momentum or Adam.
• It still multiplies Jacobians, just in the cheap order. The derivative of one layer’s output with respect to its input is a Jacobian matrix. Backpropagation multiplies a row vector (the gradient so far) by one Jacobian at a time, from the loss backward; it never forms a product of two Jacobian matrices. This order, reverse-mode automatic differentiation (Linnainmaa, 1970), is efficient whenever there are many parameters and one scalar loss; its use for training networks was popularised by Rumelhart, Hinton and Williams (1986), after Werbos (1974). “Backpropagation” is reverse-mode automatic differentiation applied to a neural network’s loss.

---

4. A Worked Example

A network with two inputs, three hidden sigmoid units and one sigmoid output, one training example, and the loss $L=\frac{1}{2}(o-t{)}^2$:

Quantity	Value
Inputs $x_1,x_2$ · target $t$	0.7, 0.9 · 0.85
Hidden weights $w_{j1},w_{j2}$ and biases $b_j$	(0.1, −0.2, 0.01), (0.3, −0.4, 0.02), (0.5, −0.6, 0.03)
Output weights $v_j$ and bias $c$ · learning rate η	0.5, −0.3, 0.2 and 0.1 · 0.01

Forward pass. Each hidden unit computes $z_j=w_{j1}{x}_1+w_{j2}{x}_2+b_j$ and $h_j=\sigma (z_j)$; the output $s={\sum }_j{v}_j{h}_j+c$ and $o=\sigma (s)$:

Unit	Weighted sum	Activation
$h_1$	$z_1=0.07-0.18+0.01=-0.10$	0.475021
$h_2$	$z_2=0.21-0.36+0.02=-0.13$	0.467546
$h_3$	$z_3=0.35-0.54+0.03=-0.16$	0.460085
$o$	$s=0.289264$	0.571816, loss $L=0.038693$

Backward pass. With ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$:

$${\delta }_o=(o-t)o(1-o)=(0.571816-0.85)(0.571816)(0.428184)=-0.068111,$$ $$\frac{\partial L}{\partial v_j}={\delta }_o{h}_j,{\delta }_j={\delta }_o{v}_j{h}_j(1-h_j),\frac{\partial L}{\partial w_{ji}}={\delta }_j{x}_i,\frac{\partial L}{\partial b_j}={\delta }_j.$$

Unit j	∂L/∂vⱼ	δⱼ = ∂L/∂bⱼ	∂L/∂wⱼ₁ · ∂L/∂wⱼ₂
1	−0.032354	−0.00849266	−0.00594486 · −0.00764340
2	−0.031845	+0.00508682	+0.00356078 · +0.00457814
3	−0.031337	−0.00338386	−0.00236870 · −0.00304548

and $\partial L/\partial c={\delta }_o=-0.068111$. Compared with central finite differences for all 13 parameters, the largest relative error was $4.5\times 10^{-9}$.

Update. Each parameter moves by $-\eta \times$ its gradient: for example $v_1=0.5-0.01\times (-0.032354)=0.500324$ and $c=0.100681$. The loss falls from 0.03869321 to 0.03861391. Every output weight increases (the output was too low), and each hidden weight moves in the direction that the sign of its $v_j$ asks for. Repeated on this one example, the output climbs to 0.731 after 1,000 steps and 0.784 after 2,000; with a learning rate of 0.01 and saturating sigmoids, learning is slow.

---

5. Making Training Work: Initialisation and Vanishing Gradients

Initial weights must differ. If all hidden units start with the same weights, they compute the same output, receive the same gradient, and stay identical forever: the layer behaves like a single unit. On the moons, 16 tanh units all started at 0.1 were still all identical after 5,000 steps, and reached 0.863 test accuracy, like a network with one hidden unit (0.859), against 0.991 from a random start. All-zero weights are worse still: with tanh, every gradient into the first layer is zero and nothing but the output bias ever moves (accuracy 0.494). Random initialisation breaks the symmetry, and its scale matters: too small and signals shrink layer by layer, too large and they explode. Glorot and Bengio (2010) and He et al. (2015) derived scales that keep the variance of activations roughly constant across layers.

Gradients can vanish. The sigmoid’s slope is at most 0.25, and close to 0 when the unit saturates. Backpropagation multiplies one such factor per layer, so in a deep sigmoid network the gradients of the first layers shrink geometrically. Measured at initialisation in a 10-layer network of width 32, the gradient of layer 1 was $7.6\times 10^{-3}$ times that of layer 10 with sigmoids, and $0.22$ with ReLU (whose slope is 1 wherever it is active) and He initialisation. This, more than anything else, is why ReLU replaced the sigmoid in hidden layers, and it is the same problem that residual connections and normalisation layers address in Going Deeper.

Train a network yourself; the “identical start” button reproduces the symmetry problem, and a learning rate above 10 the divergence:

---

6. On to Convolutional Networks

For images, fully connected layers waste parameters and ignore that nearby pixels belong together. Convolutional neural networks share small filters across the image; From Neural Networks to CNNs takes over from here. A few facts about CNNs are often stated loosely:

Statement	More precisely
”CNNs exist since 2012 (AlexNet).”	Convolutional networks trained by backpropagation date from LeCun et al. (1989; LeNet-5, 1998). AlexNet (Krizhevsky et al., 2012) made deep CNNs dominant by training a large one on GPUs.
”Output size = (W − F + 2P)/S + 1.”	Rounded down when it is not an integer: ⌊(W − F + 2P)/S⌋ + 1. A conv layer with K filters of size F × F on D input channels has K(F²D + 1) parameters.
”The number of filters should be a multiple of 2.”	Powers of two (32, 64, …) are a convention, convenient for memory layout, not a requirement.
”Softmax, tanh or another function at the end for classification.”	Softmax for one of several classes, sigmoid for two classes or independent labels. Tanh outputs lie in (−1, 1) and are not probabilities.
”A CNN recognises features independent of their location.”	Convolution is translation-equivariant (a shifted input gives a shifted feature map); pooling adds only partial invariance, and real CNNs can change their output after a shift of one pixel (Azulay & Weiss, 2019).

---

7. Common Claims, Checked

Claim	Verdict
”A neuron computes an activation function followed by a linear combination.”	The other way round: the weighted sum w·x + b first, then the activation.
”Stacking more layers makes a network more powerful.”	Only with non-linear activations: without them, 16 hidden units gave exactly logistic regression’s 0.8628.
”Backpropagation is a method of minimising gradients.”	It computes gradients (to 4.5 × 10⁻⁹ of finite differences in the example); gradient descent uses them to minimise the loss.
”Backpropagation avoids Jacobian matrices.”	It multiplies vectors by one Jacobian at a time, backward from the scalar loss (reverse-mode automatic differentiation), at about the cost of 1.8 forward passes instead of 159,020.
”Mini-batches are used because the gradient is easier to derive that way.”	The gradient is the same formula. Mini-batches trade gradient noise (falling as 1/B: 2.37 at B = 1, 0.019 at B = 128) against the number of updates and hardware efficiency.
”A larger learning rate trains faster.”	Up to a point: 1 reached a loss of 0.008, 10 got stuck at 0.189, and 100 ended at chance.
”Initial weights can be zero or all equal.”	Equal weights stay equal: 16 identical units behaved like one (0.863 vs. 0.991 with random weights); all-zero weights did not learn at all with tanh.
”Sigmoid is a good hidden activation.”	Its slope is at most 0.25: layer-1 gradients were 7.6 × 10⁻³ of layer 10’s in a 10-layer sigmoid network, 0.22 with ReLU.
”One hidden layer can learn any function.”	One hidden layer can represent any continuous function on a bounded set with enough units; how many units, how much data and whether training finds the weights are separate questions.
”ReLU is the residual linear unit.”	Rectified linear unit: max(0, z).
”RMSE = Σ(yᵢ − ŷᵢ)².”	That is the residual sum of squares. RMSE = √((1/n) Σ (yᵢ − ŷᵢ)²), in the units of y.

---

Summary

Idea	In one line
Neuron	Weighted sum, then a non-linear activation (step, sigmoid, tanh, ReLU)
Network	Layers of neurons; without non-linearity, still a linear model
Training	Minimise a loss by gradient descent; the learning rate decides speed and stability
Mini-batches	Unbiased, noisy gradients (noise ∝ 1/B), many cheap updates
Backpropagation	Chain rule backward from the loss; all gradients for ~2 forward passes
Initialisation	Random, to break symmetry, at a scale that keeps signals stable
Depth	Sigmoid gradients vanish; ReLU, careful initialisation, residual connections help

This concludes the Machine Learning series: from describing data, through regression, classification, resampling, trees and unsupervised learning, to the networks that the Computer Vision and Spiking Neural Networks series build on.

The experiments: neural_networks.py (numpy 2.1, seeded), with backpropagation written by hand and checked against finite differences. The lab computes the worked example in the browser and reproduces every number above (o = 0.571816, L = 0.038693, the same gradients); its playground’s gradients agree with finite differences to within 8 × 10⁻⁸ for ReLU, tanh and no activation.

---

References

Neurons and networks

• McCulloch, W. S., & Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5(4).
• Rosenblatt, F. (1958). The perceptron: A probabilistic model for information storage and organization in the brain. Psychological Review, 65(6).
• Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4).
• Hornik, K. (1991). Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2).
• Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. ICML.

Backpropagation and training

• Linnainmaa, S. (1970). The representation of the cumulative rounding error of an algorithm as a Taylor expansion of the local rounding errors. Master’s thesis, University of Helsinki.
• Werbos, P. J. (1974). Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. PhD thesis, Harvard University.
• Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088).
• Robbins, H., & Monro, S. (1951). A stochastic approximation method. Annals of Mathematical Statistics, 22(3).
• Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. AISTATS.
• He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. ICCV.
• Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2018). Automatic differentiation in machine learning: A survey. Journal of Machine Learning Research, 18(153).

Convolutional networks

• LeCun, Y., Boser, B., Denker, J. S., et al. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4).
• LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11).
• Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. NeurIPS.
• Azulay, A., & Weiss, Y. (2019). Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research, 20(184).

Textbooks

• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
• James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning (2nd ed.). Springer.