Efficient CNNs: Doing More With Less

The previous post was about depth: how skip connections made 50- and 152-layer networks trainable. This one is about cost. VGG16 needs 138 million parameters and 15.5 billion multiply-adds per image. By 2019, EfficientNet-B0 beat it by almost 6 points of accuracy with 26× fewer parameters and 40× less compute.

None of that came from one big idea. It came from a handful of small, reusable tricks, each attacking a specific waste:

Idea	What it saves	Introduced by
1×1 convolution	Shrinks channels before expensive layers; replaces giant fully-connected layers	Network in Network (2014)
Inception module	Lets the network choose kernel sizes, with 1×1 “bottlenecks”	GoogLeNet (2015)
Grouped convolution	Splits one wide layer into many cheap parallel paths	ResNeXt (2017)
Squeeze-and-excitation	Adds global context for almost no compute	SE-Net (2018)
Dense connectivity	Reuses features instead of relearning them	DenseNet (2017)
Depthwise separable convolution	Splits a convolution into a spatial step and a channel step: about 9× cheaper	MobileNet (2017)
Compound scaling	Grows depth, width and resolution in balance	EfficientNet (2019)

---

1. The 1×1 Convolution

A 1×1 convolution sounds useless: a filter that looks at a single pixel. But a convolution filter always spans all input channels. So at each pixel, a 1×1 convolution takes the vector of $C_{in}$ channel values and multiplies it by a $C_{in}\times C_{out}$ weight matrix:

A 1×1 convolution is a fully-connected layer applied at every pixel, with the same weights everywhere. That makes it useful in three ways:

• Changing the number of channels cheaply. Squeezing 256 channels down to 64 costs only 256 × 64 weights. Every architecture in this post uses it as a bottleneck in front of expensive layers.
• Adding non-linearity. A 1×1 convolution followed by ReLU adds depth without changing what the network sees spatially [Lin et al. 2014].
• Replacing fully-connected layers. A dense layer from $c$ inputs to $c^{\prime }$ outputs is exactly a 1×1 convolution with $c^{\prime }$ filters. Swap the classifier’s dense layers for 1×1 convolutions and the network becomes fully convolutional: it accepts any input size and outputs a map of predictions instead of a single one [Long et al. 2015]. That’s the starting point for segmentation in the next post.

Normal 3×3 convolutions relate neighbouring pixels. 1×1 convolutions relate channels. Most modern blocks alternate the two.

---

2. Inception: Let the Network Choose

What’s the right kernel size, 1×1, 3×3 or 5×5? It depends on the data, and on the layer. GoogLeNet’s answer: use all of them in parallel and let training decide what each is good for [Szegedy et al. 2015].

Every Inception module runs four branches side by side and concatenates their outputs along the channel axis:

• a 1×1 convolution,
• a 3×3 convolution,
• a 5×5 convolution,
• a 3×3 max pool. If the right feature already exists in the previous layer, the network can just pass it on instead of relearning it.

Why the 1×1 bottlenecks matter

Run 3×3 and 5×5 convolutions directly on all 192 input channels and the cost explodes. The fix is a cheap 1×1 convolution that reduces the channels first. Here is the real module 3a (input 28 × 28 × 192):

Branch	Without bottlenecks	With bottlenecks
1×1	192 × 64 = 12,288	12,288
3×3	3·3·192·128 = 221,184	192·96 + 3·3·96·128 = 129,024
5×5	5·5·192·32 = 153,600	192·16 + 5·5·16·32 = 15,872
pool	0, but passes all 192 channels on	192 · 32 = 6,144
Total weights	387,072	163,328
Output channels	416	256
Multiply-adds (× 28·28)	303M	128M

The bottlenecks cut the cost by 2.4×. They also keep the output from growing: without the pool branch’s 1×1 projection, every module would add 192 channels on top of its input, and nine stacked modules would explode.

GoogLeNet

• 9 Inception modules, 22 layers deep, and only about 7 million parameters: 12× fewer than AlexNet, while far more accurate. It needs 1.5 billion multiply-adds per image.
• It won ILSVRC 2014 with a 6.67% top-5 error.
• Two auxiliary classifiers branch off the middle of the network during training, adding their loss (weighted 0.3) so the early layers get a stronger gradient signal. They’re dropped at test time. Later work found they mostly act as a regularizer rather than speeding up training [Szegedy et al. 2016].
• Global average pooling and a single fully-connected layer at the end, instead of AlexNet’s and VGG’s giant heads.

Later versions refined the idea [Szegedy et al. 2016]: Inception v2/v3 replace each 5×5 with two 3×3 (VGG’s trick), factor an $n\times n$ convolution into $1\times n$ followed by $n\times 1$, and add batch normalization.

You can run the real GoogLeNet below, in the model zoo: it shows, for your photo, how much each of the four branches contributes in every Inception module.

---

3. ResNeXt: More Paths, Same Budget

ResNet’s bottleneck block has one path through a 64-channel 3×3 convolution. ResNeXt splits it into 32 parallel paths, each only 4 channels wide, and adds up their outputs [Xie et al. 2017]:

• The number of paths is called cardinality. The paper found that increasing cardinality helps more than making the network wider or deeper with the same budget.
• With the same number of weights (about 70K per block) and the same compute, ResNeXt-50 reaches 77.6% top-1 on ImageNet vs. 76.1% for ResNet-50.
• The 32 paths are implemented as one grouped convolution with 32 groups, the same operation AlexNet used in 2012 to split its layers across two GPUs. What was a hardware workaround became a design principle.

---

4. Squeeze-and-Excitation: Attention Over Channels

A 3×3 convolution only sees a small neighbourhood, so each channel’s response is computed without knowing what’s happening in the rest of the image. The squeeze-and-excitation (SE) block adds that global view [Hu et al. 2018]:

1. Squeeze: global average pooling reduces each of the $C$ feature maps to a single number, a summary of the whole image.
2. Excite: a tiny two-layer network (with a bottleneck of $C/16$) turns those $C$ numbers into $C$ weights between 0 and 1.
3. Scale: each feature map is multiplied by its weight. Channels that matter for this image are amplified, the rest are damped.

It’s a form of attention, over channels instead of positions, and it’s almost free: adding SE blocks to ResNet-50 costs about 10% more parameters and under 1% more compute, and cuts the top-1 error from 24.8% to 23.3%. An SE network won the last ILSVRC classification challenge in 2017 (2.25% top-5 error). EfficientNet’s building block uses it too.

The next lecture takes attention much further: Transformers apply it over positions, letting every part of the image look at every other part.

---

5. DenseNet: Reuse Instead of Relearn

ResNet adds the input to a block’s output. DenseNet concatenates instead, and connects every layer to every layer before it [Huang et al. 2017]:

• Layer $\ell$ receives the feature maps of all previous layers in its block, so the information an early layer computed never has to be recomputed. Each layer only adds a few new channels, the growth rate $k$ (e.g. 32), so the layers can be very thin.
• Between dense blocks, transition layers (a 1×1 convolution and 2×2 average pooling) shrink the channel count and the resolution.
• The result is very parameter-efficient: DenseNet-121 has 8.0M parameters (ResNet-50: 25.6M) and reaches 74.4% top-1, and DenseNet-201 (20M) matches ResNet-101 (44.5M).
• The catch: all those concatenated feature maps have to be kept in memory, so DenseNets are memory-hungry and often slower than their parameter count suggests.

---

6. MobileNet: Depthwise Separable Convolutions

A standard 3×3 convolution does two jobs at once: it looks at neighbouring pixels and it mixes channels. MobileNet splits those jobs into two cheap steps [Howard et al. 2017]:

1. Depthwise convolution: one 3×3 filter per channel, applied to that channel only. It looks at neighbours but doesn’t mix channels.
2. Pointwise convolution: a 1×1 convolution that mixes the channels. It mixes channels but doesn’t look at neighbours.

For a $k\times k$ kernel, $C$ input and $C^{\prime }$ output channels on an $H\times W$ map:

$$\underset{\text{standard}}{\underbrace{k^2\cdot C\cdot C^{\prime }\cdot HW}}\text{vs.}\underset{\text{depthwise}}{\underbrace{k^2\cdot C\cdot HW}}+\underset{\text{pointwise}}{\underbrace{C\cdot C^{\prime }\cdot HW}}\Rightarrow \frac{\text{separable}}{\text{standard}}=\frac{1}{C^{\prime }}+\frac{1}{k^2}$$

For 3×3 kernels that’s about 8–9× fewer multiply-adds. Try it:

The results:

• MobileNet v1: 70.6% top-1 with 4.2M parameters and 569M multiply-adds. VGG16 gets 71.5%, but with 32× more parameters and 27× more compute.
• Two knobs trade accuracy for speed: a width multiplier α thins every layer, and a resolution multiplier ρ shrinks the input.
• MobileNetV2 [Sandler et al. 2018] adds inverted residual blocks: expand the channels with a 1×1, run the cheap depthwise 3×3 in the expanded space, then project back down with a 1×1 without ReLU (a “linear bottleneck”). 72.0% top-1 with 3.4M parameters and only 300M multiply-adds.

A common mistake in the cost formula

You’ll sometimes see the depthwise cost written as $9\cdot W\cdot H$. That’s missing a factor of $C$: there is one 3×3 filter per channel, so the depthwise step costs $9\cdot C\cdot W\cdot H$. Without the $C$, the savings look absurdly large.

---

7. EfficientNet: Scale Everything Together

Once you have a good small network, how should you make it bigger for more accuracy? Add layers (depth)? Add channels (width)? Feed it bigger images (resolution)? Each helps, but each alone saturates quickly. EfficientNet’s insight is that they should grow together [Tan & Le 2019]:

$$\text{depth}={\alpha }^{\varphi },\text{width}={\beta }^{\varphi },\text{resolution}={\gamma }^{\varphi },\alpha \cdot {\beta }^2\cdot {\gamma }^2\approx 2$$

With α = 1.2, β = 1.1 and γ = 1.15, each step of φ roughly doubles the compute while keeping the three in balance: a bigger image needs more layers to cover it and more channels to capture its finer patterns.

• EfficientNet-B0, the starting point, was found by neural architecture search. It’s built from MobileNetV2’s inverted residual blocks with squeeze-and-excitation added.
• B1 to B7 are B0 scaled up. Slide through them:

• EfficientNet-B7 reached 84.3% top-1, matching the best model of the time (GPipe) while being 8.4× smaller and 6.1× faster.
• EfficientNet-Lite variants drop the squeeze-and-excitation blocks and use ReLU6, which makes them run well on phones with 8-bit arithmetic. That’s the version in the demo below.

---

8. The Big Picture: Run Them Side by Side

Here is every architecture from this post and the previous one, on one chart:

The best models moved up and to the left over time: more accuracy for less compute. MobileNetV2 matches VGG16’s accuracy with about 50× fewer multiply-adds, and EfficientNet-B0 beats ResNet-50 with 10× less compute. The EfficientNet curve shows that once you have an efficient base network, scaling it is a smooth trade.

And here are four of them running in your browser on the same photo. Every model below is the real network (8-bit weights from the ONNX Model Zoo):

Things to look at:

• The download sizes: GoogLeNet is 7.1 MB and MobileNetV2 3.7 MB. AlexNet was 61 MB and VGG16 138 MB.
• GoogLeNet’s branch chart: for every Inception module, how much of the output comes from the 1×1, 3×3, 5×5 and pooling branches. All four are used, and the mix changes with depth.
• The answers: on the sample photos, EfficientNet-Lite4 is both the most accurate and the most confident (93% “mountain bike”, 97% “Egyptian cat”), while being far smaller than ResNet-50.
• The times: MobileNetV2 is usually the fastest, but notice that time doesn’t follow multiply-adds exactly (see the next section).

---

9. Common Claims, Checked

Claim	What’s actually true
”A depthwise separable convolution costs $9\cdot W\cdot H+W\cdot H\cdot C\cdot C^{\prime }$.”	The depthwise part is $9\cdot W\cdot H\cdot C$: one 3×3 filter per channel. The saving is $1/C^{\prime }+1/9$, about 8–9×, not hundreds.
”Fewer parameters or fewer multiply-adds means a faster network.”	Not necessarily. Speed also depends on memory access and how well the operations fit the hardware. DenseNet’s concatenations use lots of memory, and depthwise convolutions do so little work per byte that GPUs run them inefficiently. ShuffleNet V2 [Ma et al. 2018] showed networks with the same FLOPs running at very different speeds, and argued for measuring latency directly.
”A 1×1 convolution just scales each pixel.”	It mixes all channels at each pixel: it’s a dense layer applied per pixel. That’s what makes it useful as a bottleneck and as a replacement for fully-connected layers.
”Global pooling makes networks resolution-independent.”	Correct: after global average pooling the vector size depends only on the number of channels, so ResNet, GoogLeNet and MobileNet accept any input size.
”EfficientNet (Tan & Le, 2020)” / “DenseNet (2018)“	EfficientNet was published at ICML 2019; DenseNet won the CVPR 2017 best paper award.

---

Summary

• 1×1 convolutions mix channels per pixel: cheap bottlenecks, and a drop-in replacement for fully-connected layers.
• Inception runs several kernel sizes in parallel and uses 1×1 bottlenecks to keep that affordable: GoogLeNet won ILSVRC 2014 ahead of VGG with 20× fewer parameters.
• ResNeXt splits a block into many thin parallel paths (grouped convolution) for better accuracy at the same cost.
• Squeeze-and-excitation re-weights channels using a summary of the whole image: attention over channels, for under 1% extra compute.
• DenseNet concatenates all earlier feature maps, so layers reuse features instead of relearning them.
• Depthwise separable convolutions split spatial filtering from channel mixing, making convolutions about 8–9× cheaper.
• EfficientNet scales depth, width and resolution together from a searched base network.
• Fewer parameters and multiply-adds don’t automatically mean faster: measure on the target hardware.

Next: Semantic Segmentation: Labeling Every Pixel: encoder-decoders, U-Net, dilated convolutions and DeepLab, built on the backbones from these two posts.

---

References

Building blocks

• Lin, M., Chen, Q., & Yan, S. (2014). Network in network. ICLR.
• Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. CVPR.
• Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML.

Architectures

• Szegedy, C., Liu, W., Jia, Y., et al. (2015). Going deeper with convolutions. CVPR.
• Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016). Rethinking the Inception architecture for computer vision. CVPR.
• Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks. CVPR.
• Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. CVPR.
• Huang, G., Liu, Z., van der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. CVPR.
• Howard, A. G., Zhu, M., Chen, B., et al. (2017). MobileNets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861.
• Chollet, F. (2017). Xception: Deep learning with depthwise separable convolutions. CVPR.
• Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., & Chen, L.-C. (2018). MobileNetV2: Inverted residuals and linear bottlenecks. CVPR.
• Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. ICML.
• Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. NeurIPS. (Grouped convolution across two GPUs.)
• He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. CVPR.

Measuring efficiency

• Canziani, A., Paszke, A., & Culurciello, E. (2016). An analysis of deep neural network models for practical applications. arXiv:1605.07678.
• Ma, N., Zhang, X., Zheng, H.-T., & Sun, J. (2018). ShuffleNet V2: Practical guidelines for efficient CNN architecture design. ECCV.