Classification

What the Task Does

Classification assigns one or more category labels to an input such as an image, document, or signal. It supports applications including diagnosis, content moderation, quality inspection, spam detection, and species identification.

Unlike detection or segmentation, ordinary image classification describes the input as a whole. It does not say where the evidence appears.

Core Idea

A model learns a function f(x)f(x) that maps an input xx to class scores called logits. An output layer turns those scores into probabilities or independent label scores, and a decision rule selects the prediction.

For mutually exclusive classes, softmax converts logit ziz_i into

p(y=i∣x)=exp⁡(zi)∑jexp⁡(zj).p(y=i\mid x)=\frac{\exp(z_i)}{\sum_j \exp(z_j)}.

The class with the highest probability is the usual prediction. During training, cross-entropy increases the score of the correct class relative to the alternatives.

Main Variants

Binary Classification

Binary classification separates two outcomes, such as defective and non-defective. A single sigmoid output commonly represents p(y=1∣x)p(y=1\mid x) and is trained with binary cross-entropy.

Multi-class Classification

Multi-class classification chooses one class from several mutually exclusive options, such as cat, dog, or bird. It commonly uses one logit per class, softmax, and categorical cross-entropy.

Multi-label Classification

Multi-label classification allows several labels to be true at once, as with tags describing one image. Each class receives an independent sigmoid output and usually its own binary cross-entropy term.

Type Valid output Typical activation Typical loss
Binary One of two classes Sigmoid Binary cross-entropy
Multi-class One of NN classes Softmax Categorical cross-entropy
Multi-label Any subset of NN labels Sigmoid per label Binary cross-entropy per label

Temperature scaling replaces ziz_i with zi/Tz_i/T before softmax. T<1T<1 sharpens the distribution and T>1T>1 softens it; fitting TT on validation data can improve calibration without changing class order.

How It Works

A feature extractor such as a CNN or Vision Transformer converts the input into a representation. A classification head then pools spatial features, applies one or more learned layers, and produces class logits.

Image-classification architectures illustrate the evolution of this feature extractor:

Era Architecture Main contribution
2012 AlexNet Deep CNN training on GPUs
2014 VGGNet Depth built from small 3 × 3 kernels
2015 ResNet Residual connections
2019 EfficientNet Compound depth, width, and resolution scaling
2020 Vision Transformer Self-attention over image patches
2022 ConvNeXt A modernized convolutional design

Fine-tuning a pretrained feature extractor is often more effective than training from scratch, especially when the task dataset is small.

How It Is Evaluated

Metric What it measures Useful when
Accuracy Fraction of predictions that are correct Classes and error costs are balanced
Precision Fraction of positive predictions that are correct False positives are costly
Recall Fraction of actual positives that are found False negatives are costly
F1 score Harmonic mean of precision and recall A single balance of both is needed
Top-kk accuracy Whether the correct class is among the top kk scores The label space is large

A confusion matrix compares actual and predicted classes. It reveals systematic confusions that disappear inside a single aggregate score. For imbalanced data, inspect per-class and macro-averaged metrics rather than accuracy alone.

Calibration is a separate property from accuracy. Reliability diagrams, expected calibration error, or the Brier score test whether events assigned probability pp occur approximately a fraction pp of the time.

Practical Considerations

  • Transfer learning: Start from a relevant pretrained backbone when labeled data is limited.
  • Data augmentation: Use transformations that preserve the label while reflecting deployment variation.
  • Class imbalance: Consider weighted losses, balanced sampling, focal loss, and class-specific thresholds.
  • Thresholds: Choose them from operational error costs rather than assuming 0.5 is optimal.
  • Dataset design: Prevent near-duplicates, subjects, or acquisition sessions from leaking across splits.

Limitations and Failure Modes

  • A classifier may exploit background, watermark, or acquisition artifacts instead of the intended object.
  • Closed-set models force unfamiliar inputs into known classes unless rejection or out-of-distribution handling is added.
  • Softmax values can be overconfident and are not automatically calibrated probabilities.
  • Aggregate accuracy can hide poor performance on rare classes or demographic and environmental subgroups.
  • Distribution shifts in lighting, sensors, geography, or class prevalence can invalidate validation results.

Related