Detection
What the Task Does
Object detection identifies objects of interest and estimates where each one appears. One image can produce a variable-length set of bounding boxes, class labels, and confidence scores.
Classification answers what the whole image contains. Detection answers what and where, making it useful for autonomous driving, robotics, surveillance, and industrial inspection.
Core Idea
A detector transforms an image into candidate objects. Each candidate combines a class score with a box, commonly represented by corner coordinates or center and size .
The model must solve classification and localization together while deciding how many predictions correspond to real objects. Training therefore matches predictions to annotations and optimizes both class and box losses.
Main Variants
Two-stage Detectors
Two-stage models first generate candidate regions, then classify and refine them. R-CNN, Fast R-CNN, and Faster R-CNN belong to this family. The extra proposal stage can improve control and accuracy but adds latency.
One-stage Detectors
One-stage models predict classes and boxes directly from feature maps. YOLO, SSD, and RetinaNet are examples. Their dense, single-pass computation is well suited to real-time inference.
Anchor-based and Anchor-free Detectors
Anchor-based models predict offsets from predefined reference boxes at each location. Anchor-free models instead predict centers, corners, or distances to box boundaries without a fixed anchor catalogue.
| Formulation | Mechanism | Examples |
|---|---|---|
| Two-stage | Propose regions, then classify and refine | Faster R-CNN |
| One-stage | Predict dense boxes and classes directly | YOLO, SSD, RetinaNet |
| Anchor-based | Regress from reference boxes | Faster R-CNN, YOLOv2–v5 |
| Anchor-free | Predict geometry without predefined anchors | FCOS, CenterNet, YOLOv8+ |
These categories overlap: a one-stage detector can be anchor-based or anchor-free.
How It Works
Extract and Combine Features
A backbone such as ResNet, EfficientNet, or CSPDarknet extracts visual features. A neck such as FPN, PANet, or BiFPN combines several resolutions so the detector can represent both small and large objects.
Predict Candidate Objects
The detection head outputs box coordinates, class scores, and sometimes a separate objectness score. Decoupled heads use different branches for classification and box regression.
Modern YOLO models illustrate one-stage detection. Early YOLO divided the image into a grid; later versions added multi-scale prediction, anchors, feature-pyramid necks, and then anchor-free heads.
Assign Training Targets
Label assignment decides which prediction locations or proposals learn from each ground-truth object. The loss usually combines classification with an IoU-based or coordinate-based localization term.
Remove Duplicates
Dense heads often predict several overlapping boxes for one object. Non-maximum suppression (NMS) keeps a strong candidate and removes lower-scoring boxes whose overlap exceeds a threshold. Some end-to-end detectors learn one-to-one outputs instead.
How It Is Evaluated
Intersection over Union (IoU) measures box overlap:
A prediction counts as a true positive only when its class is correct and its IoU with an unmatched ground-truth box exceeds the chosen threshold. Duplicate detections and unmatched predictions are false positives.
Average Precision (AP) summarizes a class's precision-recall curve. Mean Average Precision (mAP) averages AP across classes. mAP@0.5 uses IoU 0.5; COCO-style mAP@0.5:0.95 averages over thresholds from 0.5 to 0.95.
Production evaluation should also measure latency, throughput, memory use, and performance by object size. Report hardware, batch size, input resolution, precision, and preprocessing so speed claims are reproducible.
Practical Considerations
- Start with a pretrained detector and fine-tune it on representative target data.
- Include the expected range of object sizes, aspect ratios, occlusions, and backgrounds.
- Higher input resolution and high-resolution feature maps often help small objects, at a compute cost.
- Tune confidence and NMS thresholds against the deployment error costs.
- Check annotations carefully: inconsistent box boundaries create ambiguous localization targets.
Limitations and Failure Modes
- Small, crowded, truncated, or heavily occluded objects may not produce distinct features.
- Bounding boxes include background and cannot represent an object's exact shape.
- NMS can suppress valid overlapping objects or retain duplicate boxes when thresholds are poorly chosen.
- Rare classes and long-tailed datasets can produce large gaps between overall and per-class AP.
- Performance can fall under camera, weather, viewpoint, or scene shifts absent from training data.