Classification says what is in an image. Segmentation labels every pixel. Object detection finds each object and draws a box around it: for objects and classes, the output is , class scores plus four box coordinates per object.
Two things make this harder than it looks:
- The number of objects is unknown. An image can contain zero cars or fifty. A network has a fixed number of outputs, so detectors predict many candidate boxes and then filter them.
- Objects come in every size and shape. A detector has to find a distant pedestrian and a nearby bus in the same image.
This post follows how the field solved both, from R-CNN (2014), which took 47 seconds per image, to detectors that run on a phone in real time.
1. Is a Box Correct? Intersection over Union
Before building a detector, we need a way to say whether a predicted box is right. The standard measure is Intersection over Union (IoU): the area where the two boxes overlap, divided by the area they cover together. Drag the predicted box:
- IoU is 1 for a perfect match and 0 for no overlap.
- PASCAL VOC counts a detection as correct if IoU ≥ 0.5. COCO is stricter: it averages over thresholds from 0.5 to 0.95.
- IoU is used everywhere in detection: to decide which anchors are positives during training, to remove duplicates (NMS), and to score the result.
2. Two-Stage Detectors: The R-CNN Family
The first deep detectors split the problem in two: first find regions that might contain an object, then classify each region with a CNN.
- R-CNN [Girshick et al. 2014] used a classic algorithm (selective search) to propose about 2,000 regions, warped each one to 224 × 224, and ran a full CNN on every region. It improved the best PASCAL VOC 2012 result by more than 30% (relative), to 53.3% mAP, but took around 47 seconds per image.
- Fast R-CNN [Girshick 2015] runs the CNN once on the whole image, then cuts out each region’s features with RoI pooling (a max pool that turns any region into a fixed 7 × 7 grid). 213× faster at test time, but selective search still took about 2 seconds.
- Faster R-CNN [Ren et al. 2015] replaces selective search with a Region Proposal Network (RPN): a small network on the same features that proposes regions using anchor boxes. The whole detector became one network, running at about 5 images per second.
Anchor boxes
How can a network output “boxes” when it only knows how to output fixed-size tensors? Anchors give every location of the feature map a fixed set of reference boxes (here 3 sizes × 3 shapes), and the network predicts a class and a small correction for each:
- During training, an anchor that overlaps a real object with IoU ≥ 0.5 becomes a positive: it learns that object’s class and the offsets that turn the anchor into the exact box. All others learn “background”.
- So the network never invents a box from nothing; it picks and adjusts one from a dense, fixed set.
Mask R-CNN [He et al. 2017] adds a third head to Faster R-CNN that predicts a pixel mask for each detected box. That’s instance segmentation: separate masks for each person, not just “person pixels”. It also replaced RoI pooling with RoIAlign, which avoids rounding region coordinates to whole pixels.
3. One-Stage Detectors: YOLO and SSD
Two stages are accurate but slow. One-stage detectors predict classes and boxes for all anchors or cells in a single pass.
YOLO
YOLO (“You Only Look Once”) [Redmon et al. 2016] divides the image into a grid. Each cell predicts a few boxes, a confidence for each, and class probabilities:
YOLO v1 ran at 45 images per second (63.4% mAP on PASCAL VOC 2007), fast enough for live video. Its weakness was small objects and crowds: each cell could only predict two boxes. Later versions added anchors and predictions at several scales.
SSD
SSD [Liu et al. 2016] predicts from anchor boxes (“default boxes”) on several feature maps of different resolutions: the high-resolution early maps catch small objects, the coarse late ones catch big objects. SSD300 reached 74.3% mAP on VOC 2007 at 59 images per second.
The YOLO family, briefly
YOLO became a whole lineage, maintained by different groups:
| Version | Year | By | Main idea |
|---|---|---|---|
| YOLO | 2016 | Redmon et al. (CVPR) | One pass over a grid |
| YOLOv2 / YOLO9000 | 2017 | Redmon & Farhadi (CVPR) | Anchor boxes chosen by k-means, batch norm, multi-scale training |
| YOLOv3 | 2018 | Redmon & Farhadi (tech report) | Predictions at 3 scales, Darknet-53 backbone |
| YOLOv4 | 2020 | Bochkovskiy et al. | CSPDarknet backbone, mosaic augmentation, CIoU loss; still anchor-based |
| YOLOv5 | 2020 | Ultralytics (code, no paper) | PyTorch, easy training and export |
| YOLOX | 2021 | Ge et al. | Anchor-free, separate heads for class and box |
| YOLOv6 / YOLOv7 | 2022 | Meituan / Wang et al. | Faster backbones and training tricks; v7 adds pose estimation |
| YOLOv8 | 2023 | Ultralytics | Anchor-free; detection, segmentation, pose and classification in one framework |
| YOLOv9, v10, 11 | 2024 | various | v10 trains to need no NMS at all |
4. Non-Maximum Suppression
Any detector built this way predicts many overlapping boxes for each object: every anchor or cell near the object fires. Non-maximum suppression (NMS) turns them into one detection per object:
- Sort the boxes by score.
- Keep the highest one.
- Remove every remaining box of the same class that overlaps it by more than a threshold (typically IoU > 0.5).
- Repeat with the next highest box that’s left.
Watch two real detectors do it
Below, SSD (SSDLite with a MobileNetV2 backbone) and Tiny-YOLOv3 run in your browser. The page shows the raw boxes they predict, then applies the score threshold and NMS itself, so you can see every step:
Things to try:
- Switch to “before NMS”: every object is covered by a stack of near-duplicate boxes.
- Read the counts: on the bicycle photo, SSD scores all 1,917 of its anchors; about 16 pass the score threshold, and NMS leaves 2: one person, one bicycle.
- Drag “NMS overlap” up to 0.95, so that only nearly identical boxes are suppressed: the duplicates come back (10 boxes for 2 objects). Drag it very low and boxes of neighbouring objects start suppressing each other (try the dog walker).
- Tiny-YOLOv3’s grid: tick show the grid cells that fire to see which cells produced each box. Large objects come from the coarse 13 × 13 grid, smaller ones (the dog) from the finer 26 × 26 grid.
- The same mistakes again: on the dog walker, SSD reports a horse and a sheep. COCO-trained detectors, like the classifiers in the earlier posts, sometimes mistake fluffy dogs for farm animals.
5. RetinaNet and the Focal Loss
For years, two-stage detectors were clearly more accurate than one-stage ones. Lin et al. asked why, and found the main reason wasn’t the second stage itself but class imbalance [Lin et al. 2017b].
A one-stage detector evaluates around 100,000 candidate boxes per image, and almost all of them are easy background. Each easy background box has a small cross-entropy loss, but there are so many that together they dominate training. Two-stage detectors avoid this because their proposal step already throws away most of the background.
The fix is the focal loss, which multiplies the cross-entropy by , where is the probability the model gives the correct class:
- With , an easy example with contributes 100× less than with cross-entropy, while a hard example barely changes. Training concentrates on the boxes that are actually difficult.
- RetinaNet combines the focal loss with a Feature Pyramid Network [Lin et al. 2017a], which adds a top-down path so that every resolution level has strong, meaningful features. RetinaNet was the first one-stage detector to match two-stage accuracy (39.1 AP on COCO).
6. FCOS: Detection Without Anchors
Anchors work, but they bring many design choices: how many sizes, which shapes, which IoU counts as a match. FCOS [Tian et al. 2019] drops them entirely. Every location inside an object directly predicts its distances to the four sides of the box :
- Locations near the edge of an object tend to produce poor boxes. FCOS adds a center-ness output, 1 at the centre and falling toward the edges, and multiplies it into the score so that NMS prefers boxes from central locations.
- Like RetinaNet, FCOS predicts on several pyramid levels, assigning objects of different sizes to different levels.
- Anchor-free designs are now standard: YOLOX, YOLOv8 and later all predict boxes this way.
7. Measuring a Detector: Precision, Recall and mAP
A detector makes two kinds of mistakes: false positives (a box where there’s no object, or a duplicate) and false negatives (a missed object). Sorting all detections by confidence and walking down the list traces a precision-recall curve:
- Precision: of the boxes predicted so far, how many are correct?
- Recall: of all real objects, how many have been found so far?
Average precision (AP) is the area under that curve, for one class. Click the detections to change which are right:
- mAP is AP averaged over all classes. On PASCAL VOC it’s computed at IoU 0.5 (“mAP@0.5”).
- COCO AP additionally averages over IoU thresholds from 0.50 to 0.95 in steps of 0.05, so it rewards boxes that fit tightly, not just roughly. That’s why COCO numbers (around 40) look much lower than VOC numbers (70–80) for the same detector.
- A high-confidence false positive hurts more than a low-confidence one: it pulls precision down early in the list.
8. Where Detection Went Next
- DETR [Carion et al. 2020] treats detection as predicting a set: a Transformer outputs a fixed number of boxes, and training matches them one-to-one with the real objects. No anchors, and no NMS, because each object is assigned to exactly one prediction. It’s the bridge to the next lecture on attention and Transformers.
- YOLOv10 [Wang et al. 2024] brought the same idea to YOLO: trained with one-to-one assignment, it no longer needs NMS at inference.
9. Common Claims, Checked
| Claim | What’s actually true |
|---|---|
| ”YOLOv4 introduced an anchor-free detection head.” | YOLOv4 is anchor-based. Anchor-free YOLO heads came with YOLOX (2021) and YOLOv8 (2023). |
| ”YOLOv5 (2021) … YOLOv7 (2023)“ | YOLOv5 was released in 2020 (code only, no paper); YOLOv7 appeared in 2022 (published at CVPR 2023). The series continued with v9, v10 and 11 in 2024. |
| ”Two-stage detectors are more accurate because they have two stages.” | The RetinaNet paper showed the main cause is class imbalance: the proposal stage filters out most easy background. With the focal loss, a one-stage detector matched two-stage accuracy. |
| ”FCOS predicts the distances to the center of the object.” | Each location predicts its distances to the four sides of the box (l, t, r, b). Center-ness is a separate score for how central that location is. |
| ”Anchor boxes are 1:1, 1:2 and 2.1.” | The usual aspect ratios are 1:1, 1:2 and 2:1 (Faster R-CNN: 3 ratios × 3 scales = 9 anchors per location). |
| ”A detection is correct if its box overlaps the object.” | It must overlap enough: IoU ≥ 0.5 for PASCAL VOC, and COCO averages over 0.5–0.95. A duplicate of an already matched object counts as a false positive. |
Summary
- Detectors predict many candidate boxes and filter them: by score, and by NMS to remove duplicates.
- IoU measures box overlap; it decides training positives, duplicates and correctness.
- Two-stage detectors (R-CNN → Fast → Faster R-CNN) propose regions, then classify them; Faster R-CNN made the whole thing one network with anchors and an RPN. Mask R-CNN adds per-object masks.
- One-stage detectors (YOLO, SSD) predict everything in one pass, fast enough for video.
- Focal loss fixed one-stage detectors’ real weakness, class imbalance; FPN gave every scale strong features.
- Anchor-free detectors like FCOS predict distances to the box sides directly; most modern YOLOs do too.
- Detectors are compared by mAP: area under the precision-recall curve, averaged over classes (and, on COCO, over IoU thresholds).
This completes the tour of CNN architectures: classification backbones, efficient designs, segmentation and detection. Next: Attention from Scratch, the idea DETR, SE-Net and PSPNet were all reaching for: letting every part of an image look at every other part.
References
Two-stage detectors
- Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR.
- Girshick, R. (2015). Fast R-CNN. ICCV.
- Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. NeurIPS.
- He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask R-CNN. ICCV.
One-stage and anchor-free detectors
- Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. CVPR.
- Redmon, J., & Farhadi, A. (2017). YOLO9000: Better, faster, stronger. CVPR.
- Redmon, J., & Farhadi, A. (2018). YOLOv3: An incremental improvement. arXiv:1804.02767.
- Bochkovskiy, A., Wang, C.-Y., & Liao, H.-Y. M. (2020). YOLOv4: Optimal speed and accuracy of object detection. arXiv:2004.10934.
- Ge, Z., Liu, S., Wang, F., Li, Z., & Sun, J. (2021). YOLOX: Exceeding YOLO series in 2021. arXiv:2107.08430.
- Wang, C.-Y., Bochkovskiy, A., & Liao, H.-Y. M. (2023). YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. CVPR.
- Wang, A., Chen, H., Liu, L., et al. (2024). YOLOv10: Real-time end-to-end object detection. NeurIPS.
- Liu, W., Anguelov, D., Erhan, D., et al. (2016). SSD: Single shot multibox detector. ECCV.
- Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017a). Feature pyramid networks for object detection. CVPR.
- Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017b). Focal loss for dense object detection. ICCV.
- Tian, Z., Shen, C., Chen, H., & He, T. (2019). FCOS: Fully convolutional one-stage object detection. ICCV.
- Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-end object detection with Transformers. ECCV.
Benchmarks
- Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010). The PASCAL visual object classes (VOC) challenge. IJCV, 88, 303–338.
- Lin, T.-Y., Maire, M., Belongie, S., et al. (2014). Microsoft COCO: Common objects in context. ECCV.