Going Deeper: From AlexNet to ResNet

The previous post explained how a single convolutional layer works. This one follows what happened when people started stacking many of them. Between 2012 and 2015, ImageNet error dropped from 26% to under 4%, mostly by making networks deeper. Depth also caused new problems, and solving them gave us the most influential idea in modern deep learning: the skip connection.

The post covers:

1. The four basic vision tasks
2. What a CNN actually learns, including what it shouldn’t
3. Building blocks the architectures rely on: global pooling, dropout, batch normalization
4. AlexNet and VGG: deeper is better
5. Why you can’t just keep stacking layers: vanishing gradients and the degradation problem
6. ResNet: how skip connections fixed it, and why they work
7. Common claims about these networks, checked against the papers

---

1. Four Tasks, One Backbone

Computer vision has four basic pattern recognition tasks. They differ in how much of the image gets a label:

Task	Question	Output for a $W\times H$ RGB image
Classification	What is in the image?	$N$ class scores
Semantic segmentation	Which class is each pixel?	$W\times H\times N$
Object detection	Where is each object, and what is it?	$M\times (N+4)$: for $M$ objects, $N$ class scores plus a box $(x,y,w,h)$
Instance segmentation	Which pixels belong to each object?	A mask per object, e.g. $M\times W\times H$

Here $N$ is the number of classes and $M$ the number of detected objects.

Here are all four, running live in your browser on the same image. Pick a photo (or use your own; it never leaves your device) and press Run:

Notice how the output grows richer from task to task: one label, then a class for every pixel, then boxes that separate objects, then a mask per object.

These are small models built to run on phones, and their mistakes are worth looking at too. On the dog-walker photo the detector calls one dog a sheep and another a horse, and the classifier’s top guess is “horse cart”. On the cat, the classifier is unsure (5% for “Christmas stocking”), while segmentation and detection both find the cat. The instance segmentation model here (BodyPix) only knows people; general instance segmentation needs a model like Mask R-CNN, which is too heavy to run comfortably in a browser.

All four use the same kind of backbone: a CNN that turns pixels into features. The architectures in this post were designed for classification, but they’re also the backbones inside segmentation networks (U-Net, DeepLab) and detectors (Faster R-CNN, RetinaNet, Mask R-CNN). Get the backbone right and every task improves.

---

2. What Does a CNN Actually Learn?

Edges first, objects later

Train a CNN on natural images and look at its first-layer filters. They turn into oriented edge detectors and colour-contrast blobs [Krizhevsky et al. 2012; Zeiler & Fergus 2014]. Nobody programs this; it comes out of training. These filters look a lot like Gabor filters and like the “simple cells” Hubel & Wiesel found in the visual cortex [Hubel & Wiesel 1962].

Deeper layers combine those edges into textures, then parts (eyes, wheels), then whole objects [Zeiler & Fergus 2014]. The first layers are so generic that they transfer almost unchanged between datasets, while the last layers are specific to the training task [Yosinski et al. 2014]. That is why fine-tuning a pretrained network works so well.

You can see all of this inside a real network. The explorer below runs MobileNet v2 (trained on ImageNet) on a photo and shows its first-layer filters, its feature maps at five depths, and a class activation map (CAM) of where it looks [Zhou et al. 2016]:

Things to try:

• Filters tab: pick a filter that looks like a light/dark split and watch it outline the cars and people. Pick a solid-colour filter and it lights up regions of that colour instead.
• Depth tab: step from layer 1 to last conv. The maps go from 112×112 sharp edge images to a 7×7 grid of blobs. Each step trades where information for what information.
• Where it looks: on the street photo, the CAM for “Irish wolfhound” sits on the dog, not the man or the cars. On the bicycle, “mountain bike” sits on the bike. Try your own photos to look for shortcuts.

…and sometimes the wrong thing

A CNN learns whatever separates the classes in the training data, and that isn’t always the object. In a well-known example, a husky-vs-wolf classifier learned “snow in the background = wolf” [Ribeiro et al. 2016]. Classifiers trained on camera-trap images fail on cows standing on a beach, because cows “live” on grass [Beery et al. 2018].

So a classifier trained on airplane photos (mostly blue sky behind them) can call a dog photographed against blue water an airplane. Accuracy on the test set won’t warn you, because the test set has the same bias. This is called shortcut learning [Geirhos et al. 2020], and it is one reason to look at what a network attends to, not just its accuracy. Class activation maps like the one above, and their gradient-based successor Grad-CAM [Selvaraju et al. 2017], are the standard first check.

---

3. Building Blocks

Besides convolution, stride and pooling (covered in the previous post), three more building blocks show up in every architecture below. Each one solves a specific problem.

Global average pooling: from feature maps to one vector

Problem: at the end of the convolutional layers you have a stack of feature maps, say $7\times 7\times 512$. The classifier needs a single vector. The old way was to flatten the maps ($7\cdot 7\cdot 512=25,088$ numbers) and feed them into a dense layer. That layer is enormous, and it only works for one input size.

Solution: average each feature map into one number [Lin et al. 2014]:

• $7\times 7\times 512$ becomes a vector of 512 numbers, one per channel. Each number answers “how strongly was this feature present anywhere in the image?”
• It has no weights at all, while VGG’s flatten + dense layer has 102.8 million.
• A $7\times 7$ map and a $10\times 10$ map both give 512 numbers, so the same network accepts any image size. ResNet, MobileNet and most modern CNNs end this way.

Dropout: training an ensemble inside one network

Problem: a big network can memorize the training set. Neurons learn to rely on specific partners (“co-adaptation”) instead of learning features that are useful on their own.

Solution: during training, switch off each neuron at random with probability $p$, choosing a new random set at every step [Srivastava et al. 2014]:

• No neuron can count on any other being there, so each has to learn something useful by itself. That reduces overfitting.
• With $n$ droppable neurons there are $2^n$ possible “thinned” networks, all sharing weights. Training with dropout trains all of them at once; testing with every neuron on approximates averaging their predictions. That’s why dropout is described as an ensemble inside one model, similar to how a random forest trains each tree on a random subset of the features.
• At test time nothing is dropped. To keep the expected size of the activations the same, the survivors are scaled up by $\frac{1}{1-p}$ during training (×2 for $p=0.5$), so the network needs no change at inference. This is called inverted dropout, and it’s what PyTorch and TensorFlow do.

Batch normalization: keeping every layer in a healthy range

Problem: each layer’s input is the previous layer’s output. If the weights are slightly too small, activations shrink a little at every layer; slightly too large, and they grow. Over many layers that compounds into activations that vanish or explode, and training stalls.

Solution: for each channel, normalize the activations over the mini-batch, then let the network rescale them with two learned parameters $\gamma$ and $\beta$ [Ioffe & Szegedy 2015]:

$$\hat{x}=\frac{x-{\mu }_{\text{batch}}}{\sqrt{{\sigma }_{\text{batch}}^2+ϵ}},y=\gamma \hat{x}+\beta$$

Run it on a 30-layer network and compare the activation size at every layer, with and without batch normalization:

Without batch norm, the result depends entirely on getting the initialization exactly right: weights about 3× too small and activations shrink to about $10^{-14}$ by layer 30; less than 2× too large and they grow to about $10^7$. With batch norm, even the bad initialization stays at a steady ~0.6 at every layer.

That’s why batch norm let Inception reach the same accuracy in 14× fewer training steps, with much higher learning rates. It also acts as a mild regularizer, because each example is normalized with statistics from a random mini-batch.

Training vs. inference

During training, $\mu$ and $\sigma$ come from the current mini-batch. At inference there may be only one image, so batch norm uses running averages of $\mu$ and $\sigma$ collected during training instead. Forgetting to switch modes (model.eval() in PyTorch) is a classic bug.

---

4. AlexNet (2012): Depth + GPUs

AlexNet won the 2012 ImageNet challenge (ILSVRC) with a top-5 error of 15.3%, against 26.2% for the runner-up [Krizhevsky et al. 2012]. Top-5 error means the right class wasn’t among the model’s five best guesses. That gap of almost 11 percentage points convinced the field to switch to deep learning.

Depth	8 layers with weights: 5 convolutional + 3 fully-connected
Size	60 million parameters, 650,000 neurons
Hardware	Trained for 5–6 days on two GTX 580 GPUs with 3 GB each. The model was split across both GPUs because it didn’t fit on one
What was new	ReLU instead of tanh (about 6× faster training), dropout in the fully-connected layers, heavy data augmentation (random crops, flips, colour shifts), overlapping max pooling, local response normalization

The paper’s main lesson: depth matters. Removing any one of the middle convolutional layers cost about 2% top-1 accuracy.

The architecture

Read it left to right:

• The image shrinks while the channels grow: 227 → 55 → 27 → 13 → 6 pixels across, and 3 → 96 → 256 → 384 channels. Pooling after conv1, conv2 and conv5 does most of the shrinking.
• Big filters first, small ones later: 11×11 with stride 4 in conv1 cuts the resolution by 4× in one step, which was needed to fit the GPUs of 2012. Later networks replaced this with stacks of 3×3 (see VGG).
• The two GPUs: each conv block is split in half (dashed line), and the halves only talk to each other at conv3 and in the fully-connected layers.
• Where the parameters are: the five conv layers have only 2.3M parameters. The three fully-connected layers have 58.6M, 96% of the network, and fc6 alone has 37.8M.

Run the real AlexNet

Below, the original AlexNet weights (the Caffe reference model, quantized to 8 bits, 61 MB) run in your browser. Press Load AlexNet & run, then click any layer in the strip to see what it computed:

Things to look at:

• conv1 shows the network’s 96 learned 11×11 filters. One group of 48 learned colour blobs, the other group learned grey edge detectors at every angle. The paper reported exactly this split between its two GPUs. Below the filters you can see their responses: outlines of people, dogs and cars.
• conv2 → conv5: the maps get coarser and respond to larger structures instead of edges.
• fc6: only about 1 in 7 of the 4,096 neurons is active for a given image. The rest are exactly 0 after ReLU.
• fc8 on the cat photo: AlexNet says schipperke (a small black dog breed) with 99% confidence. It’s confidently wrong, unlike MobileNet in the demo above, which was unsure. A high probability is not the same as a correct answer.

---

5. VGG (2014): Simplicity and Depth

VGG asked a simple question: what if we use only 3×3 convolutions and just go deeper [Simonyan & Zisserman 2015]?

• VGG16 has 13 convolutional layers + 3 fully-connected layers. VGG19 has 16 + 3.
• Every convolution is 3×3, and nothing else changes from layer to layer except the number of channels. The whole design fits in one sentence.
• VGG took 2nd place in classification and 1st in localization at ILSVRC 2014 (7.3% top-5 error).

Why only 3×3? Stacking small filters

AlexNet started with 11×11 and 5×5 filters. VGG replaces every large filter with a stack of 3×3 filters. Follow one output pixel back through two 3×3 layers:

• The output pixel depends on a 3×3 patch of the layer below, and each of those 9 pixels depends on its own 3×3 window of the input. Together the windows cover 5×5, the same area as one 5×5 filter. Three stacked layers cover 7×7.
• The stack is cheaper: 2 × 9 = 18 weights per channel pair instead of 25, and 27 instead of 49 for 7×7.
• It’s more expressive: there’s a ReLU between the two layers, so the stack can model a non-linear function of the 5×5 patch, which a single 5×5 filter can’t.

This idea, that depth plus small filters beats a few large filters, shaped almost every CNN after VGG. (More on receptive fields in the previous post.)

The architecture

VGG16 is five blocks of 3×3 convolutions, each followed by a 2×2 max pool:

• Only the pools shrink the image. With padding 1, a 3×3 convolution keeps the size, so the resolution halves exactly five times: 224 → 112 → 56 → 28 → 14 → 7.
• Channels double after each pool, from 64 up to 512, so each layer does roughly the same amount of work.
• The receptive field grows with every layer: from 3×3 after the first convolution to 212×212 after the last pool, almost the whole image.

Run the real VGG16

The model below is the real VGG16 (from GluonCV, quantized to 8 bits). Notice the two buttons: VGG is 138 MB, but because 89% of it is the fully-connected layers, the page can download just the first 15 MB (all 13 conv layers) and show you everything they compute. Only when you ask for a prediction does it fetch the remaining 124 MB.

Things to look at:

• conv1_1: its 64 filters are only 3×3 pixels, much less recognizable than AlexNet’s 11×11 filters, but their responses are the same kind of edge maps.
• Step through the conv layers: each layer’s description gives its receptive field, which grows from 3×3 to 196×196 at conv5_3. The maps become coarser and respond to larger and larger structures.
• The download sizes: 15 MB for 13 conv layers, then 124 MB for 3 fully-connected layers. That’s the next section’s point, made in your network tab.
• Mistakes, again: VGG16 calls the golden dog a Pembroke (a corgi breed), the black cat a schipperke again (80%), and the dog walker’s puffy coat a fur coat (69%).

The cost: 138 million parameters

VGG16 has 138.4 million parameters, and VGG19 has 143.7 million. What surprises most people is where those parameters are:

The first fully-connected layer alone, flattening $7\times 7\times 512$ features into 4,096 neurons, has 102.8 million parameters. That’s three-quarters of the whole network. The convolutional layers have only 11% of the weights but do 99% of the computation, because they run at every pixel position. So:

• Memory and overfitting come from the fully-connected head. That’s why later networks replace it with global average pooling.
• Speed depends on the convolutions. That’s what efficient architectures (MobileNet, EfficientNet) attack, as the next post will show.

Depth was already hard to train

The VGG authors couldn’t train their deepest networks from scratch. They first trained the 11-layer version, then used its weights to initialize the deeper ones. That was a hint of the problem the rest of this post is about.

Recap: overfitting

A model overfits when it has more capacity than the data can justify. It learns details of the training set that don’t generalize, so it scores well on training data and poorly on the validation and test sets. Common remedies:

• More data, or data augmentation to fake more data (crops, flips, colour jitter)
• Fewer trainable parameters, e.g. replacing big fully-connected layers, or fine-tuning only the head of a pretrained model
• Regularization: weight decay, dropout, batch normalization
• Early stopping: stop training when validation loss starts to rise

---

6. Why Not Just Keep Stacking Layers?

If 8 layers beat shallow models and 19 beat 8, why not 50 or 100? Two different problems get in the way.

Problem 1: vanishing and exploding gradients

Backpropagation computes the gradient for an early layer with the chain rule, so it multiplies one factor per layer on the way back from the loss:

$$\frac{\partial \mathcal{L}}{\partial {\mathbf{h}}_1}=\frac{\partial \mathcal{L}}{\partial {\mathbf{h}}_L}\cdot \frac{\partial {\mathbf{h}}_L}{\partial {\mathbf{h}}_{L-1}}\cdots \frac{\partial {\mathbf{h}}_2}{\partial {\mathbf{h}}_1}$$

• If those factors are typically below 1, the product shrinks exponentially with depth, and the first layers barely learn. These are vanishing gradients [Hochreiter 1991; Bengio et al. 1994]. The sigmoid’s derivative is at most 0.25, so 50 sigmoid layers can scale the gradient by something like $0.25^{50}\approx 10^{-30}$.
• If they’re typically above 1, the product explodes, and weight updates become huge and unstable. These are exploding gradients [Pascanu et al. 2013].

Both come from depth itself: the problem is the number of factors in the product. Floating-point underflow is only the very last symptom.

Try it below. The playground builds a 50-layer network and measures how big the gradient is when it reaches each layer:

Typical output:

Setup	Gradient at layer 1, relative to layer 50
Sigmoid, Xavier init	about $10^{-31}$: vanished
ReLU, weights too small	about $10^{-13}$: vanished
ReLU, weights too large	about $10^7$: exploded
ReLU, He init	about $0.4$: healthy

The fixes came before ResNet:

• ReLU, whose derivative is exactly 1 for positive inputs [Nair & Hinton 2010]
• Careful initialization that keeps the variance of the signal constant from layer to layer: Xavier for tanh and sigmoid [Glorot & Bengio 2010], He for ReLU [He et al. 2015]
• Batch normalization [Ioffe & Szegedy 2015]
• Gradient clipping for exploding gradients [Pascanu et al. 2013]

Problem 2: the degradation problem

With ReLU, He initialization and batch normalization, gradients stay healthy. Yet deeper plain networks still got worse. He et al. trained plain networks with 20 and 56 layers on CIFAR-10 [He et al. 2016a]. The 56-layer network had a higher error, on the training set too.

The same happened on ImageNet (top-1 error, lower is better):

Depth	Plain network	ResNet
18 layers	27.94%	27.88%
34 layers	28.54% (worse than 18)	25.03% (better than 18)

This is called degradation, and it is easy to misdiagnose:

• It isn’t overfitting. An overfitting model does better on training data. Here, training error goes up.
• It isn’t (mainly) vanishing gradients. These networks used batch normalization, and the authors checked that the gradients reaching early layers had healthy sizes.

The ResNet paper calls it an optimization difficulty: deep plain networks are simply much harder for SGD to fit, possibly because they converge exponentially slowly.

---

7. ResNet (2015): Learning the Difference

A thought experiment

Take a network that already solves the task well, and add a few extra layers on top. The best those layers could do is nothing, passing their input straight through (the identity function). So the deeper network can always be at least as good as the shallow one.

In practice it ends up worse, which means SGD has trouble learning even the identity function with a stack of non-linear layers. ResNet’s answer: build the identity in, and let the layers learn only the change [He et al. 2016a].

The residual block

Instead of asking two layers to learn a mapping $H(x)$ directly, a residual block learns $F(x)=H(x)-x$ and adds the input back through a shortcut:

$$\text{output}=\text{ReLU}(F(x)+x)$$

• If the block’s best move is to change nothing, it only has to push its weights toward 0. That’s much easier than building an identity out of non-linear layers.
• Each block learns a correction on top of what earlier layers already computed: the residual.
• The shortcut adds no parameters and almost no compute.

Why it fixes training: a gradient highway

Differentiate the block’s output with respect to its input:

$$\frac{\partial (x+F(x))}{\partial x}=1+\frac{\partial F}{\partial x}$$

The 1 means the gradient from later layers passes through every block unchanged, on top of whatever flows through the layers. Stack many blocks and the gradient reaching the first layer always includes a direct, un-shrunk term from the loss [He et al. 2016b]:

$$\frac{\partial \mathcal{L}}{\partial x_l}=\frac{\partial \mathcal{L}}{\partial x_L}\left(1+\frac{\partial }{\partial x_l}\sum _{i=l}^{L-1}F(x_i)\right)$$

ResNets behave like ensembles of shallow networks

Veit et al. “unravelled” a residual network. Each block offers two routes (through $F$ or along the shortcut), so $n$ blocks create $2^n$ paths of different lengths through the network [Veit et al. 2016]. Their experiments showed:

• Deleting a single block from a trained ResNet at test time barely changes accuracy. Doing the same to VGG destroys it. ResNets don’t depend on one long chain.
• Most of the gradient comes from short paths. In a 110-layer ResNet, the paths that actually carry the gradient during training are only about 10–34 layers deep.

So a ResNet acts like a large ensemble of relatively shallow networks that share weights. The paths aren’t fully independent, though, so “ensemble-like” is more accurate than “an ensemble of independent classifiers”.

The bottleneck block

ResNet-18 and 34 use the two-layer block shown above. The deeper versions (50, 101, 152) use a cheaper bottleneck block:

• A 1×1 convolution squeezes 256 channels down to 64, the 3×3 convolution does its spatial work on the cheap 64-channel version, and a second 1×1 expands back to 256 so the shortcut can be added.
• That’s about 70K weights per block, against 1.2M for two plain 3×3 convolutions on 256 channels. More on 1×1 convolutions in the next post.
• When a block halves the resolution and doubles the channels, $x$ and $F(x)$ no longer have the same shape. The shortcut then becomes a 1×1 convolution with stride 2 (a projection shortcut) to match them.

The architecture

• Four stages of 3, 4, 6 and 3 bottleneck blocks. Each stage halves the resolution (56 → 28 → 14 → 7) and doubles the channels (256 → 2048), like VGG’s blocks.
• Batch normalization after every convolution (folded into the weights in the model below).
• Global average pooling and a single fully-connected layer at the end, instead of VGG’s three giant ones. That’s why ResNet-50 has only 25.6M parameters: at 8 bits it’s 26 MB, less than half of AlexNet, while being 6× deeper.

Run the real ResNet-50, and switch blocks off

The real ResNet-50 (from GluonCV, 8-bit) runs below. Besides the usual layer-by-layer view, you can do something you can’t do to AlexNet or VGG: switch residual blocks off while the network is running. A switched-off block outputs its input unchanged, as if $F(x)=0$, and the prediction updates.

Things to try:

• The bar chart shows how large each block’s update $F(x)$ is compared to its input $x$. In this network the updates aren’t tiny: a typical block adds about 60–90% of its input’s size, and the blocks in the last stage add up to 3–4× more.
• Switch one block off, or press Remove a random block. The network rarely falls apart. Across the four sample photos, removing a single block kept the original answer in the top 3 in 35 of 48 cases. Often the new answer is a close neighbour: the street dog goes from “Afghan hound” to “Saluki” or “Ibizan hound”, other sighthound breeds.
• Switch off one of the last blocks (stage 4) and the answer usually breaks (“broom”, “mobile home”). Those are the blocks with the biggest updates in the bar chart.
• Projection blocks can’t be removed: their shortcut changes the shape, so there is nothing to fall back on.

Veit et al. ran this experiment systematically on much deeper ResNets (110 and 200 layers), where each block matters less: deleting one block barely changed accuracy, while deleting one layer from VGG destroyed it. ResNet-50, with only 16 blocks, shows the same graceful behaviour in a milder form.

The results

• ILSVRC 2015 winner: 3.57% top-5 error with an ensemble, better than the roughly 5% estimated for humans on this benchmark.
• ResNet-152 is 8× deeper than VGG19 but cheaper: 11.3 billion multiply-adds per image vs. 19.6 billion, and 60 million parameters vs. 144 million.
• On CIFAR-10, the authors even trained a 1,202-layer ResNet. It reached almost zero training error, so optimization was no longer the problem. Its test error (7.9%) was worse than the 110-layer version (6.4%), but that’s ordinary overfitting on a small dataset, not degradation.

---

8. The Path So Far

Model	Year	Layers	Parameters	ImageNet top-5 error
AlexNet	2012	8	60M	15.3%
VGG16	2014	16	138M	7.3% (VGG ensemble)
GoogLeNet (Inception)	2014	22	≈7M	6.7%
ResNet-152	2015	152	60M	3.6% (ensemble)

Error fell more than 4× in three years while depth grew 19×. Parameter count didn’t follow depth: GoogLeNet and ResNet show that how you spend parameters matters more than how many you have. That’s the topic of the next post.

---

9. Common Claims, Checked

These statements come up often in tutorials and lecture notes. Here’s what the original papers say.

Claim	What’s actually true
”Very deep plain networks get worse because of vanishing gradients.”	Not according to the ResNet paper. With batch normalization the gradients were healthy, and training error still rose. It’s an optimization difficulty [He et al. 2016a].
”Gradients vanish because they get smaller as the model converges, until they drop below floating-point precision.”	They vanish because the chain rule multiplies one factor per layer, and factors below 1 shrink the product exponentially with depth. Underflow is only the extreme end of that.
”VGG uses 11 convolutional layers and has 133 million parameters.”	VGG16 has 13 conv layers and 138M parameters, and VGG19 has 16 conv layers and 144M. The 133M figure belongs to the 11-layer version (VGG-A).
”VGG is huge because of all those convolutions.”	89% of VGG16’s parameters are in its 3 fully-connected layers. The convolutions dominate compute, not size.
”The unravelled view of ResNets (Wu et al.) shows they are ensembles of independent classifiers.”	The unravelled view is from Veit et al. 2016. The paths share weights and aren’t independent; the network behaves like an ensemble of shallow networks.
”AlexNet takes 224 × 224 images.”	That’s what the paper says, but with an 11×11 filter and stride 4, $(224-11)/4+1=54.25$ isn’t a whole number. Caffe’s reference model uses 227 × 227 crops (giving 55 × 55, as in the paper’s figure); the ONNX release used above takes 224 and drops the remainder (54 × 54).
”A 3×3 convolution turns an 8×8 map into a 7×7 map.”	Without padding, it becomes 6×6 ($8-3+1$). With “same” padding it stays 8×8.
”Dropout deactivates random neurons.”	Only during training. At test time every neuron is active, and the scaling is handled so that expected activations match.

---

Summary

• The four vision tasks (classification, semantic segmentation, detection, instance segmentation) share the same kind of CNN backbone.
• First-layer filters become edge detectors, and deeper layers build parts and objects. Networks can also learn shortcuts like “blue background = airplane”.
• AlexNet proved deep CNNs work (ReLU, dropout, GPUs). VGG showed that small 3×3 filters stacked deep work even better, at the price of 138M parameters, 89% of them in the fully-connected head.
• Depth brings vanishing and exploding gradients (fixed by ReLU, good initialization, batch norm) and the degradation problem: deeper plain networks train worse, even with healthy gradients.
• ResNet adds identity shortcuts so each block learns a residual $F(x)$. The $+1$ in the gradient creates a highway back to early layers, the network behaves like an ensemble of shallow paths, and 152 layers train easily.

Next: Debugging a Fine-Tuning Script: ResNet-50 on Three Images: a pretrained ResNet-50 put to work, a notebook that fails, and every bug in it measured and fixed.

---

References

Architectures

• Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. NeurIPS.
• Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. ICLR.
• Szegedy, C., Liu, W., Jia, Y., et al. (2015). Going deeper with convolutions. CVPR.
• He, K., Zhang, X., Ren, S., & Sun, J. (2016a). Deep residual learning for image recognition. CVPR.
• He, K., Zhang, X., Ren, S., & Sun, J. (2016b). Identity mappings in deep residual networks. ECCV.
• Veit, A., Wilber, M., & Belongie, S. (2016). Residual networks behave like ensembles of relatively shallow networks. NeurIPS.

Training deep networks

• Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen (Diploma thesis). TU München.
• Bengio, Y., Simard, P., & Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166.
• Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. AISTATS.
• Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. ICML.
• Pascanu, R., Mikolov, T., & Bengio, Y. (2013). On the difficulty of training recurrent neural networks. ICML.
• He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. ICCV.
• Jia, Y., Shelhamer, E., Donahue, J., et al. (2014). Caffe: Convolutional architecture for fast feature embedding. ACM Multimedia. (Source of the reference AlexNet weights used in the demo.)
• Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML.
• Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. JMLR, 15, 1929–1958.
• Lin, M., Chen, Q., & Yan, S. (2014). Network in network. ICLR.

What CNNs learn

• Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1), 106–154.
• Zeiler, M. D., & Fergus, R. (2014). Visualizing and understanding convolutional networks. ECCV.
• Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). How transferable are features in deep neural networks? NeurIPS.
• Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I trust you?”: Explaining the predictions of any classifier. KDD.
• Beery, S., Van Horn, G., & Perona, P. (2018). Recognition in terra incognita. ECCV.
• Geirhos, R., Jacobsen, J.-H., Michaelis, C., et al. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2, 665–673.
• Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., & Torralba, A. (2016). Learning deep features for discriminative localization. CVPR.
• Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. ICCV.
• Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018). MobileNetV2: Inverted residuals and linear bottlenecks. CVPR.