Notes

Semantic Segmentation: Labeling Every Pixel

14 min read

A classifier answers what is in this image? with one label. Semantic segmentation answers it for every pixel: this pixel is road, that one is car, those are dog. For a image with classes, the output is a volume of scores.

That’s hard for the networks from the previous posts, because they were built to throw away where in favour of what. By the last layer, a ResNet has shrunk a 224 × 224 image to a 7 × 7 grid. Segmentation needs both at once: deep layers to know what a pixel belongs to, and fine resolution to know exactly where the boundary runs.

Every architecture in this post is a different answer to that tension:

ArchitectureHow it gets “what” and “where”
FCN (2015)Classifier made fully convolutional, coarse output upsampled, plus skips from earlier layers
Encoder-decoder / U-Net (2015)A learned decoder grows the resolution back, with skip connections at every level
PSPNet (2017)Pools context at several scales so each pixel knows its surroundings
DeepLab (2015–2018)Dilated convolutions keep the resolution high without losing the receptive field
nnU-Net (2021)Configures a U-Net automatically for each new dataset

1. From Classifier to Segmenter: FCN

The first step is almost free. As the previous post showed, a fully-connected layer is the same as a 1×1 convolution. Replace a classifier’s dense layers with 1×1 convolutions and it becomes fully convolutional: it accepts any image size, and instead of one vector of class scores it outputs a grid of them, one per region [Long et al. 2015].

That grid is coarse: 1/32 of the input resolution for a standard ResNet or VGG. FCN upsamples it back to full size, and recovers some detail by adding predictions from earlier, finer layers (FCN-16s uses pool4, FCN-8s also pool3). FCN-8s reached 62.2% mean IoU on PASCAL VOC 2012, a big jump at the time.

See it live

Below, two real segmentation networks run in your browser on the same photo. The FCN here is torchvision’s modern version: a ResNet-50 that uses dilated convolutions (section 5) to keep its output at 1/8 resolution instead of 1/32. Tick “show the raw 1/8-resolution output” to see the coarse grid it actually predicts before upsampling.

Things to try:

  • The raw output: on the street photo, FCN’s real prediction is a 48 × 32 grid of blocky cells. Bilinear upsampling turns it into the smooth mask, but it can’t add detail that isn’t there: look at the dog’s legs.
  • Hover: compare what the two models say pixel by pixel. They agree on most of the picture and disagree at the edges.
  • Switch DeepLab to Cityscapes. It suddenly sees vegetation, sidewalk, road and pole, because those are its classes. And it labels the dog as a person: Cityscapes has no dog class, so the model has to pick something it knows. A segmentation network can only answer with the classes it was trained on.
  • The dog walker: FCN repeats the classifiers’ mistake from the earlier posts and paints some of the dogs as horse and sheep.

2. Encoder-Decoder: Learning to Upsample

FCN upsamples in one big jump. An encoder-decoder instead mirrors the encoder with a decoder that grows the resolution back step by step, with learned layers at each step:

224×224×3112×112×6456×56×12828×28×25614×14×5127×7×51214×14×51228×28×25656×56×128112×112×64224×224×21encoder: downsample, more channelsdecoder: upsample back to full sizebottleneckThe encoder turns pixels into meaning ("high on semantics") but loses detail: the bottleneck is only 7 × 7. The decoder brings theresolution back with learned upsampling and outputs one score per class for every pixel (here 21 PASCAL classes).

How do you upsample?

The encoder shrinks with strided convolutions and pooling. The decoder needs the opposite:

  • Fixed interpolation (nearest-neighbour or bilinear): no weights, smooth but can’t invent detail.
  • Transposed convolution (sometimes called “deconvolution”, though it doesn’t undo a convolution): a learned upsampling. Each input value stamps a weighted copy of the kernel onto a larger output:
input 2 × 2output 5 × 5 (3×3 kernel, stride 2)11111121111211224221121111211uneven overlap: 1, 2 or 4 contributionsEach input value "stamps" a copy of the 3×3 kernel onto the output, stride 2 apart; where stamps overlap they add up. Even with all-onesinput and kernel, the output alternates 1, 2, 4: the source of checkerboard artifacts. Output size: (n − 1)·s + k − 2·padding.

When the kernel size isn’t divisible by the stride, the stamps overlap unevenly, and the output tends to show a faint checkerboard pattern [Odena et al. 2016]. A common fix is to resize first (bilinear), then apply a normal convolution, which overlaps evenly.

3. U-Net: Skip Connections at Every Level

The encoder-decoder has a problem: everything has to squeeze through the 7 × 7 bottleneck, so fine detail is gone by the time the decoder starts. U-Net passes it around the bottleneck [Ronneberger et al. 2015]:

224 × 224 × 64224 × 224 × 64112 × 112 × 128112 × 112 × 12856 × 56 × 25656 × 56 × 25628 × 28 × 51228 × 28 × 51214 × 14 × 1024encoder (contracting path)decoder (expanding path)skip connections: copy & concatenateEach decoder level gets the upsampled coarse features from below and the sharp features from the same level of the encoder, so it canplace boundaries precisely. (Sizes shown for a padded U-Net on a 224 × 224 input; the original used unpadded convolutions.)
  • Encoder (left): two 3×3 convolutions per level, then 2×2 max pooling; channels double each time.
  • Decoder (right): upsample, then concatenate the feature maps from the same encoder level, then two 3×3 convolutions. The encoder’s sharp, early features tell the decoder exactly where edges are; the upsampled deep features tell it what they belong to.
  • U-Net was designed for biomedical images, where labelled data is scarce. Its electron-microscopy experiments used just 30 annotated training images, relying on heavy data augmentation (especially random elastic deformations), and it won the ISBI 2015 cell-tracking challenge.

U-Net became the default architecture for medical segmentation and, ten years later, is still inside many diffusion image generators.

4. The Context Problem: PSPNet

Look at a small patch of a boat on a lake and a car on a road: up close, they look alike. A network that only sees a local neighbourhood labels the boat as a car. The surrounding context (lots of water) is what gives it away [Zhao et al. 2017].

PSPNet adds a pyramid pooling module on top of the backbone:

feature mapfrom the CNNpool to 1×11×1 convupsamplepool to 2×21×1 convupsamplepool to 3×31×1 convupsamplepool to 6×61×1 convupsamplethe original features are kept tooconcatenate+ conv → labelsPooling at 1×1 (the whole image), 2×2, 3×3 and 6×6 gives every pixel a summary of its surroundings at four scales. A boat on water and acar on a road look alike up close; the pooled context ("lots of water around") is what tells them apart.
  • The feature map is average-pooled into grids of 1×1 (the whole image), 2×2, 3×3 and 6×6.
  • Each pooled grid goes through a 1×1 convolution (to reduce channels), is upsampled back, and is concatenated with the original features.
  • So every pixel’s final prediction sees a summary of the image at four scales. PSPNet won the 2016 ImageNet scene-parsing challenge and reached 85.4% mean IoU on PASCAL VOC 2012.

5. Dilated (Atrous) Convolution

Every pooling or strided layer halves the resolution, and segmentation needs that resolution back. Why not just remove the last few downsampling steps? Because the receptive field would shrink: the deep layers would no longer see enough of the image.

Dilated (or atrous, French for “with holes”) convolution solves this. It spreads a 3×3 kernel’s 9 taps apart by a rate , so it covers a area with the same 9 weights [Chen et al. 2015; Yu & Koltun 2016]. Remove the downsampling, dilate the following layers to compensate, and you keep the large receptive field and the high resolution. Try it:

  • With one layer, rate 2 sees 5 × 5 and rate 4 sees 9 × 9, still with 9 weights.
  • Stack layers with growing rates (1, 2, 4) and the receptive field grows exponentially with depth, with every pixel covered.
  • Stack layers with the same rate > 1 and you get gridding: a regular pattern of pixels that are never looked at [Wang et al. 2018]. That’s why practical designs mix rates.

This is exactly what the FCN in the live demo does: its ResNet-50 replaces the strides of the last two stages with dilation 2 and 4, so its output is 1/8 of the input instead of 1/32.

6. DeepLab

The DeepLab family built segmentation around dilated convolutions:

  • DeepLab v1/v2 [Chen et al. 2015, 2018] used dilated convolutions in the backbone and refined the blurry output with a conditional random field (CRF), a separate step that snaps labels to image edges.
  • DeepLab v3 [Chen et al. 2017] dropped the CRF and added atrous spatial pyramid pooling (ASPP): the same feature map goes through dilated convolutions at several rates in parallel, plus a branch that pools the whole image:
feature mapoutput stride 161×1 conv3×3, rate 63×3, rate 123×3, rate 18image poolingwhole imageconcatenate+ 1×1 convThe same 3×3 kernel with rates 6, 12 and 18 looks at the features at three scales at once, with no extra weights per tap; image poolingadds the global context. (Dots show where the 9 weights of each branch land, to scale.)
  • ASPP is PSPNet’s idea (context at several scales) done with dilated convolutions instead of pooling. DeepLab v3 reached 85.7% mean IoU on PASCAL VOC 2012, and DeepLab v3+ [Chen et al. 2018b] added a small U-Net-style decoder for sharper boundaries (89.0%).

About the DeepLab in the demo

The TensorFlow.js DeepLab above is a mobile version: a MobileNetV2 backbone with a reduced ASPP that keeps only the 1×1 and image-pooling branches. It’s much smaller and faster than the full model, and less accurate.

7. UNet++ and nnU-Net

  • UNet++ [Zhou et al. 2018] replaces U-Net’s plain skip connections with nested, densely connected ones (the DenseNet idea from the previous post), so the encoder and decoder features are fused gradually instead of in one jump.
  • nnU-Net (“no new U-Net”) [Isensee et al. 2021] doesn’t change the architecture at all. Its insight is that a U-Net’s success depends mostly on dozens of configuration choices: patch size, network depth, resampling, normalization, augmentation, loss. nnU-Net computes a fingerprint of each new dataset (image sizes, spacing, intensities, class balance) and sets all of those choices automatically with rules learned from many datasets. Out of the box, it beat most specialized, hand-tuned solutions on 23 public biomedical segmentation datasets, which makes it a very hard baseline: a new architecture has to beat a well-configured U-Net, not a badly tuned one.

8. Measuring a Segmentation

How good is a segmentation? The obvious answer, pixel accuracy (the share of correctly labelled pixels), is misleading, because most pixels are usually background. Drag the predicted mask around, or press predict nothing:

  • A model that predicts nothing at all still gets about 90% pixel accuracy here, because the object covers only 42 of 400 pixels.
  • Intersection over Union (IoU), also called the Jaccard index, only looks at the object: the overlap between prediction and truth, divided by their union. Predict nothing and it’s 0%.
  • Mean IoU (mIoU) averages the IoU over all classes, so rare classes count as much as common ones. It’s the standard benchmark number (all the ”% mean IoU” figures above).
  • Dice (the F1 score of pixels) ranks models the same way as IoU, but is always higher. It’s the standard in medical imaging.

The same imbalance affects training. The usual loss is per-pixel cross-entropy, which is dominated by easy background pixels. Common fixes are weighting classes by rarity, or training directly on a soft version of the Dice score, the Dice loss [Milletari et al. 2016].

9. Common Claims, Checked

ClaimWhat’s actually true
”U-Net has a very high number of parameters and needs a lot of data.”U-Net was designed for small datasets: its electron-microscopy experiments used 30 training images, relying on heavy augmentation. Its roughly 31M parameters are about the same as a ResNet-50.
”Atrous convolution reduces the network’s invariance to small shifts.”The invariance (and the lost detail) comes from downsampling. Removing downsampling keeps the resolution; dilated convolution’s job is to restore the receptive field that removing it would cost.
”A transposed convolution is a deconvolution: it reverses a convolution.”It doesn’t invert anything. It’s a learned upsampling: the same operation a convolution uses to pass gradients backwards. “Deconvolution” in signal processing means something else.
”Pixel accuracy is a good way to measure segmentation.”With imbalanced classes, predicting only background can score 90%+. Use (mean) IoU or Dice.
”DeepLab v3 (CVPR 2017)“DeepLab v3 is an arXiv report (2017); DeepLab v3+ was published at ECCV 2018.

Summary

  • Segmentation needs what (deep, coarse features) and where (fine resolution) at the same time; every architecture balances the two.
  • FCN makes a classifier fully convolutional and upsamples its coarse score map, with skips from finer layers.
  • Encoder-decoders learn to upsample; transposed convolutions do it with weights but can cause checkerboard artifacts.
  • U-Net adds skip connections at every level and works with very little data. It’s still the medical-imaging default.
  • PSPNet and ASPP add multi-scale context; dilated convolutions keep resolution high without shrinking the receptive field (watch out for gridding).
  • nnU-Net shows that careful configuration of a plain U-Net beats most new architectures.
  • Measure with IoU / mIoU or Dice, not pixel accuracy.

Next: Object Detection: Finding What and Where: finding objects and drawing boxes around them, with anchors, IoU, non-maximum suppression, R-CNN, YOLO, RetinaNet and FCOS.

References

Architectures

  • Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. CVPR.
  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. MICCAI.
  • Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017). Pyramid scene parsing network. CVPR.
  • Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2015). Semantic image segmentation with deep convolutional nets and fully connected CRFs. ICLR.
  • Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018). DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE TPAMI, 40(4), 834–848.
  • Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017). Rethinking atrous convolution for semantic image segmentation. arXiv:1706.05587.
  • Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018b). Encoder-decoder with atrous separable convolution for semantic image segmentation. ECCV.
  • Zhou, Z., Siddiquee, M. M. R., Tajbakhsh, N., & Liang, J. (2018). UNet++: A nested U-Net architecture for medical image segmentation. DLMIA (MICCAI workshop).
  • Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18, 203–211.

Building blocks

  • Yu, F., & Koltun, V. (2016). Multi-scale context aggregation by dilated convolutions. ICLR.
  • Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., & Cottrell, G. (2018). Understanding convolution for semantic segmentation. WACV.
  • Odena, A., Dumoulin, V., & Olah, C. (2016). Deconvolution and checkerboard artifacts. Distill.
  • Milletari, F., Navab, N., & Ahmadi, S.-A. (2016). V-Net: Fully convolutional neural networks for volumetric medical image segmentation. 3DV.

Datasets

  • Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010). The PASCAL visual object classes (VOC) challenge. IJCV, 88, 303–338.
  • Cordts, M., Omran, M., Ramos, S., et al. (2016). The Cityscapes dataset for semantic urban scene understanding. CVPR.
  • Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., & Torralba, A. (2017). Scene parsing through ADE20K dataset. CVPR.

Graph View

Backlinks