From Neural Networks to CNNs: Why Images Need Convolutions

Before getting into Transformers, 3D vision, sensor fusion or generative models, it helps to have the foundation every later topic builds on: the Convolutional Neural Network (CNN).

The post covers:

1. A short neural network recap
2. Why plain fully-connected networks fail on images, with the numbers
3. How a CNN works: convolution, stride, padding, output sizes and pooling
4. What the basic story leaves out, including the limitations of CNNs

---

1. Neural Network Recap

From regression to neural networks

Start with a regression problem: predict a house price from its size and its distance to the city center.

Size (m²)	Distance (m)	Price (€)
100	…	1200
170	…	1750
50	…	800
75	…	950

Linear regression learns $\hat{y}={\mathbf{w}}^{\top }\mathbf{x}+b$. A neural network stacks several of these linear maps and puts a non-linear activation $\sigma$ between them:

$$\mathbf{z}=\sigma (W_1\mathbf{x}+{\mathbf{b}}_1),\hat{y}=W_2\mathbf{z}+b_2$$

The missing piece: non-linearity

Without $\sigma$, two stacked linear layers collapse into one: $W_2(W_1\mathbf{x})=(W_2{W}_1)\mathbf{x}$. The activation function (ReLU, sigmoid, tanh, …) gives a network its expressive power. With it, even one hidden layer can approximate any continuous function [Cybenko 1989; Hornik et al. 1989]. ReLU, $\max (0,x)$, is the default in modern CNNs because it trains much faster than saturating activations [Nair & Hinton 2010; Krizhevsky et al. 2012].

Try it: regression to neural network in Python

The playground below runs real Python (NumPy) in your browser. Edit the code and press Run.

• 1 · Linear regression fits the house-price table with gradient descent. Try lr = 0.01 (too slow) and lr = 1.1 (diverges).
• 2 · Neural network fits a curve that a straight line can’t fit. It uses one hidden ReLU layer, with backpropagation written out by hand. Set USE_RELU = False and the network’s loss drops to exactly the straight-line loss, which is the collapse described above.

Two questions to answer

Every neural network project comes down to two questions:

1. How do we define the structure (architecture)? An engineer designs it: the number of layers, their types, their widths and how they connect. Different problems have produced different families of architectures: MLPs, CNNs, RNNs and Transformers.

2. How do we set the weights to fit the training data? The weights are initialized once. Then the training loop runs over and over: predict, measure the error, compute gradients, update.

Step	What it does	Typical choices
Initialization	Sets the starting weights so signals neither vanish nor explode	Xavier/Glorot [Glorot & Bengio 2010], He/Kaiming for ReLU [He et al. 2015]
Loss function	Measures how wrong the predictions are	MSE (regression), cross-entropy (classification)
Backpropagation	Computes $\partial \mathcal{L}/\partial w$ for every weight with the chain rule	[Rumelhart et al. 1986]
Optimizer	Uses the gradients to update the weights	SGD with momentum, Adam [Kingma & Ba 2015]

---

2. Why Fully-Connected Networks Fail on Images

The small MLP above works well for house prices. Now give it an image.

The numbers

Take a modest 800 × 600 RGB image and a single hidden layer with 1,000 neurons:

$$\#\text{inputs}=800\times 600\times 3=1,440,000$$ $$\#{\text{weights}}_{H_1}=1,440,000\times 1,000=1.44\times 10^9$$

At 4 bytes per float32 weight:

$$1.44\times 10^9\times 4\text{B}\approx 5.8\mathbf{GB}$$

What the basic calculation leaves out

The 5.8 GB figure underestimates the real cost:

Memory item	Size
Weights	5.8 GB
Gradients (one per weight)	5.8 GB
Adam optimizer state (2 moments per weight)	11.5 GB
Total for one layer	≈ 23 GB

That’s before activations, the batch dimension, or any further layers. And memory isn’t the only problem:

1. Statistical inefficiency. 1.44 billion free parameters need an enormous amount of labeled data to avoid overfitting.
2. No notion of space. An MLP takes the image as a flat vector. If you shuffle the pixels with a fixed permutation, it learns just as well, because it ignores the fact that neighboring pixels are related.
3. No translation equivariance. A cat in the top-left corner and a cat in the bottom-right corner activate completely different weights. The network has to learn “cat” separately for every position.

The historical answer: hand-crafted features

Before deep learning, vision pipelines avoided this problem by engineering features by hand:

Typical hand-crafted features were Gabor filters [Daugman 1985], SIFT [Lowe 2004] and HOG [Dalal & Triggs 2005], fed into a classifier such as an SVM [Cortes & Vapnik 1995]. These features were clever but brittle: each new domain needed new engineering.

Deep learning removes the split. The network learns the features itself, and the convolutional layer is what makes that tractable.

---

3. How a CNN Works

The basic idea

CNNs build in three assumptions (inductive biases) about images:

1. Locality. Useful patterns such as edges, corners and textures are small and local, so each neuron only looks at a small patch.
2. Weight sharing. A pattern detector that works in one place works everywhere, so the same small filter is reused across the whole image.
3. Hierarchy. Simple patterns combine into complex ones (edges → textures → parts → objects), so layers are stacked.

These ideas go back to Hubel & Wiesel’s discovery of local receptive fields in the visual cortex [Hubel & Wiesel 1962], then the Neocognitron [Fukushima 1980], and finally the first CNNs trained with backpropagation [LeCun et al. 1989; LeCun et al. 1998].

Local vs. global search

A convolutional layer does a local search for features: each output only depends on a small neighborhood. A dense (fully-connected) layer does a global search: it assumes every pixel could contribute to every feature. For images the local assumption is almost always right. The pixels that make up an eye sit next to each other.

The 2D convolution operation

Let $X=(x_{i,j})$ be an $n\times m$ input, $\kappa =({\kappa }_{i,j})$ a $k\times l$ kernel and $b$ a bias. The output is an $(n-k+1)\times (m-l+1)$ matrix with entries:

$$(X\ast \kappa {)}_{i,j}=\sum _{u=0}^{k-1}\sum _{v=0}^{l-1}{x}_{i+u,j+v}\cdot {\kappa }_{u+1,v+1}+b$$

In words, slide the kernel over the image and compute a weighted sum at each position.

A detail that's often left out: it's actually cross-correlation

Mathematical convolution flips the kernel ($x_{i-u,j-v}$). The formula above, and every deep-learning library (torch.nn.Conv2d, TensorFlow’s Conv2D), skips the flip, which makes it cross-correlation. Because the kernel is learned, the difference doesn’t matter: the network just learns the flipped kernel [Goodfellow et al. 2016, ch. 9].

Multi-channel convolution

The single-matrix formula hides how real CNNs handle channels. An RGB image has $C_{in}=3$ channels, and later feature maps have dozens or hundreds:

• One kernel has shape $k\times k\times C_{in}$. It spans all input channels and produces one output channel.
• A layer with $C_{out}$ kernels produces $C_{out}$ output channels.
• Parameters per layer: $k\cdot k\cdot C_{in}\cdot C_{out}+C_{out}$

Compare this with the MLP:

Layer	Parameters	Depends on image size?
Dense, 800×600×3 → 1,000	1,440,001,000	Yes
Conv 3×3, 3 → 64 channels	3·3·3·64 + 64 = 1,792	No

That’s about 800,000× fewer parameters, and the count stays the same whether the image is 32×32 or 4K. This is the main reason CNNs made deep learning on images practical.

2D vs. 3D convolution

• 2D convolution: the kernel slides along height and width and spans all channels. This is used for ordinary images.
• 3D convolution: the kernel also slides along a depth axis. This is used for volumetric data (MRI/CT scans), video (time as the third axis) and hyperspectral imaging.

The key difference is the kernel depth. In 2D convolution the kernel is always as deep as the input (here all 3 colour channels), so it can’t move in that direction and each filter produces a flat map. In 3D convolution the kernel is shallower than the volume, so it also moves through depth and each filter produces a volume. That captures patterns across neighbouring MRI slices or video frames, at a higher cost: a 3×3×3 kernel has 3× the weights of a 3×3 one per channel.

Stride

By default the kernel moves 1 pixel at a time. With stride $(s_1,s_2)$ it skips pixels, which shrinks the output:

$$\text{output}=\lfloor \frac{n-k}{s_1}\rfloor +1\times \lfloor \frac{m-l}{s_2}\rfloor +1$$

The floor matters. If the stride doesn’t divide $(n-k)$ evenly, the last rows and columns on the right and bottom are silently ignored.

Example: a 3×3 filter with stride 2 on a 5×5 input gives $\lfloor (5-3)/2\rfloor +1=2$, so a 2×2 output.

Padding

Every convolution shrinks the feature map: a $k\times k$ kernel only fits in $(n-k+1)$ positions per row. To keep the size, add a border of zeros around the input (zero padding). Here is a 4×4 input with a 3×3 kernel of ones, so each output is just the sum of the numbers under the kernel:

Some things to notice:

• Without padding, 4×4 becomes 2×2. With one ring of zeros, the output stays 4×4.

• The middle four outputs (10, 11, 9, 10) are the same in both cases. Padding only adds outputs at the border.

• The border outputs are smaller (the corners are 4) because part of their window is zeros. Zero padding slightly darkens the edges of a feature map.

• Without padding, the corner pixel (1) falls inside only one window, while a centre pixel falls inside four. Padding lets border pixels take part in as many windows as the others, so information at the edges isn’t underused.

• “Valid” padding means no padding, so the output shrinks.

• “Same” padding means enough padding that the output has the same size as the input (for stride 1). With an odd kernel size $k$, pad $(k-1)/2$ on each side.

• With an even kernel size, the padding can’t be symmetric. Usually one fewer row or column of zeros goes on the top and left than on the bottom and right.

The general output-size formula

$$H_{out}=\lfloor \frac{H_{in}+P_{top}+P_{bottom}-K_H}{S_H}\rfloor +1W_{out}=\lfloor \frac{W_{in}+P_{left}+P_{right}-K_W}{S_W}\rfloor +1$$

The number of output channels is simply the number of filters.

Worked example

	Value
Input $[N,H,W,C]$	$[100,255,255,64]$
Kernel $[K_H,K_W,C]$	$[7,7,64]$
Stride $[S_H,S_W]$	$[3,4]$
Padding (all sides)	2
Number of filters	128

$$H_{out}=\frac{255+2+2-7}{3}+1=\frac{252}{3}+1=85$$ $$W_{out}=\frac{255+2+2-7}{4}+1=\frac{252}{4}+1=64$$

Output: $[100,85,64,128]$. The batch size passes through unchanged.

Some things to notice:

• Both divisions are exact, so no pixels are dropped. A stride of 5 would give $\lfloor 252/5\rfloor +1=51$ and drop 2 rows.
• The kernel depth (64) must match the input channels (64). This is a common source of shape errors.
• Parameters: $7\cdot 7\cdot 64\cdot 128+128=401,536$.
• Unequal strides (3 vs. 4) turn a square input into a non-square output (85 × 64).

Here’s how you’d check it in PyTorch, which uses the channels-first layout $[N,C,H,W]$:

import torch
import torch.nn as nn
 
x = torch.randn(100, 64, 255, 255)          # [N, C, H, W]
conv = nn.Conv2d(in_channels=64, out_channels=128,
                 kernel_size=7, stride=(3, 4), padding=2)
 
print(conv(x).shape)                          # torch.Size([100, 128, 85, 64])
print(sum(p.numel() for p in conv.parameters()))  # 401536

PyTorch can’t run in the browser, so here is the same layer written in NumPy, which you can run. It builds the convolution from the definition: pad the input, cut out every 7×7 window with the stride applied, then take the weighted sum over all 64 input channels. It checks the output shape, the parameter count and one output value computed by hand. Try stride = (5, 5) to see rows get dropped.

Max pooling

Max pooling takes the maximum over each window of the feature map. With a 2×2 window and stride 2:

┌───────┬───────┐
│ 1   4 │[6]  2 │              ┌───┬───┐
│ 3  [9]│ 2   1 │   max pool   │ 9 │ 6 │
├───────┼───────┤  ─────────►  ├───┼───┤
│ 5  [8]│ 4   3 │   2×2, s=2   │ 8 │ 5 │
│ 1   2 │[5]  2 │              └───┴───┘
└───────┴───────┘

Watch it step by step below. Press Play or Step, then change the window size and stride. With a 3×3 window and stride 1 the windows overlap. With a 5×5 input, a 2×2 window and stride 2, the last row and column are never covered, which is the floor in the output-size formula at work.

Properties:

• No learnable parameters. It’s a fixed operation.
• A 2×2 window with stride 2 halves height and width, so there are 4× fewer activations.
• Pooling can also use padding and stride, and the same output-size formula applies.
• It keeps the strongest response in each region: “Was this feature present somewhere here?”
• It gives local translation invariance. If a feature moves by one pixel inside the window, the output stays the same. In face recognition, the eyes are never in exactly the same pixel, and pooling makes the network tolerant of that.

Putting it together: a typical CNN

As you go deeper, the spatial size shrinks (pooling and stride) while the number of channels grows (more filters). The network trades “where” information for “what” information.

---

4. What the Basic Story Leaves Out

Receptive field

One 3×3 conv sees a 3×3 patch. Stacking layers grows the receptive field: $L$ stacked 3×3 convolutions with stride 1 see a $(2L+1)\times (2L+1)$ region.

This is why VGG used only 3×3 kernels [Simonyan & Zisserman 2015]. Two 3×3 layers have the same receptive field as one 5×5 layer, with fewer parameters and an extra non-linearity:

Option	Receptive field	Parameters (C → C channels)
One 5×5 conv	5×5	$25C^2$
Two 3×3 convs	5×5	$18C^2$ (28% less)
One 7×7 conv	7×7	$49C^2$
Three 3×3 convs	7×7	$27C^2$ (45% less)

One caveat: the effective receptive field is much smaller than the theoretical one. Pixels near the center have far more influence than pixels at the edge [Luo et al. 2016].

Equivariance vs. invariance

These two words are easy to mix up:

• Convolution is translation equivariant. Shift the input and the feature map shifts by the same amount.
• Pooling (and global pooling at the end) adds invariance. Shift the input and the output stays (roughly) the same.

Pooling isn’t as robust as often claimed

Pooling is often described as making CNNs robust to rotations and small changes. In practice that holds only partly:

• Rotation: CNNs are not rotation invariant by design. Any rotation robustness comes from data augmentation, not from pooling.
• Translation: even small 1-pixel shifts can change a modern CNN’s prediction, because striding and pooling cause aliasing [Azulay & Weiss 2019]. Anti-aliased pooling partly fixes this [Zhang 2019].
• Pooling isn’t required: strided convolutions can replace pooling with no loss in accuracy [Springenberg et al. 2015].

Limitations of CNNs

These limitations explain why the field moved beyond plain CNNs:

Limitation	Why it matters	What addresses it
Local receptive field	Relating distant parts of an image takes many layers	Attention / Vision Transformers [Vaswani et al. 2017; Dosovitskiy et al. 2021]
Texture bias	ImageNet-trained CNNs rely on texture more than shape [Geirhos et al. 2019]	Shape-biased training, Transformers, augmentation
Need for labeled data	Supervised CNNs need millions of labels	Self-supervised learning
Built for 2D grids	Point clouds (LiDAR) are unordered sets, not grids	PointNet and other point-based architectures
No built-in rotation or scale invariance	Objects at new angles or sizes may be missed	Augmentation, feature pyramids, equivariant networks
Discriminative only	Classifies images but can’t generate them	GANs, diffusion models

Milestone CNN architectures

The next post covers these in detail:

Year	Architecture	Key idea
1998	LeNet-5 [LeCun et al. 1998]	Conv + pooling + dense; digit recognition
2012	AlexNet [Krizhevsky et al. 2012]	ReLU, dropout, GPU training; won ImageNet by a large margin
2014	VGG [Simonyan & Zisserman 2015]	Depth with uniform 3×3 convolutions
2015	Batch Norm [Ioffe & Szegedy 2015]	Normalizes activations; stabilizes deep training
2016	ResNet [He et al. 2016]	Skip connections allow networks with 100+ layers

---

Summary

• Neural networks are stacked linear maps with non-linearities, trained by initialization → loss → backpropagation → optimizer.
• Fully-connected networks don’t scale to images. One hidden layer on an 800×600 image needs 1.44 billion weights (about 23 GB with gradients and Adam state), and it ignores spatial structure.
• CNNs fix this with locality, weight sharing and hierarchy. A 3×3 conv layer needs about 1,800 parameters regardless of image size.
• Output size = $\lfloor (\text{in}+\text{padding}-\text{kernel})/\text{stride}\rfloor +1$, and the output channels equal the number of filters.
• Max pooling downsamples and gives local translation invariance, but CNNs are not truly shift or rotation invariant.
• CNN limitations (local context, texture bias, need for labeled data, grid-only input) motivate modern computer vision: Transformers, self-supervised learning, point-cloud networks and generative models.

Next: Going Deeper: From AlexNet to ResNet, where stacking many of these layers leads to vanishing gradients, the degradation problem and skip connections.

---

References

Foundations of neural networks

• Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323, 533–536.
• Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4), 303–314.
• Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2(5), 359–366.
• Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. AISTATS.
• Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. ICML.
• He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. ICCV.
• Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. ICLR.
• Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press. (Chapter 9: Convolutional Networks)

Hand-crafted features

• Daugman, J. G. (1985). Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters. Journal of the Optical Society of America A, 2(7), 1160–1169.
• Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297.
• Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.
• Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. CVPR.

Origins of CNNs

• Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1), 106–154.
• Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4), 193–202.
• LeCun, Y., Boser, B., Denker, J. S., et al. (1989). Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4), 541–551.
• LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324.

CNN architectures and techniques

• Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. NeurIPS.
• Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. ICLR.
• Springenberg, J. T., Dosovitskiy, A., Brox, T., & Riedmiller, M. (2015). Striving for simplicity: The all convolutional net. ICLR Workshop.
• Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML.
• He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. CVPR.
• Dumoulin, V., & Visin, F. (2016). A guide to convolution arithmetic for deep learning. arXiv:1603.07285.

CNN analysis and limitations

• Luo, W., Li, Y., Urtasun, R., & Zemel, R. (2016). Understanding the effective receptive field in deep convolutional neural networks. NeurIPS.
• Azulay, A., & Weiss, Y. (2019). Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research, 20(184), 1–25.
• Zhang, R. (2019). Making convolutional networks shift-invariant again. ICML.
• Geirhos, R., Rubisch, P., Michaelis, C., Rauber, J., Wichmann, F. A., & Brendel, W. (2019). ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR.

Beyond CNNs

• Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. NeurIPS.
• Dosovitskiy, A., Beyer, L., Kolesnikov, A., et al. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. ICLR.