Convolution

What Convolution Does

Convolution turns an image or feature tensor into maps of local pattern responses. A learned detector is applied at every spatial position, so one set of weights can find the same feature wherever it appears.

This preserves spatial organization while avoiding the parameter cost of connecting every input pixel to every output. In CNNs, convolution is the main operation used to build edges, textures, parts, and higher-level features.

Core Idea

A small tensor of weights, called a kernel, moves across the input. At each location, the kernel and the overlapping input patch are multiplied element by element and summed to produce one output value.

Deep-learning libraries usually implement discrete cross-correlation while calling it convolution. For input XX, kernel KK, and bias bb, one output channel is

Yi,j=∑u,v,cKu,v,c Xi+u,j+v,c+bY_{i,j} = \sum_{u,v,c} K_{u,v,c}\,X_{i+u,j+v,c} + b

Mathematical convolution flips the kernel before applying it. That distinction does not reduce a learned layer's capacity because training can learn the correspondingly flipped weights.

How It Works

Scan Local Regions

At each output location, the kernel covers a local window and computes a dot product. Moving the kernel across the input produces a feature map: high or low values indicate how strongly the local content aligns with the learned pattern.

The same weights are reused at every location. This weight sharing lets a layer detect a pattern independently of its position and keeps its parameter count independent of image width and height.

Combine Input and Output Channels

For an input with CinC_{in} channels, each standard kernel spans all input channels. A bank of CoutC_{out} kernels produces CoutC_{out} feature maps, so the weight tensor has shape

Cout×Cin×Kh×Kw.C_{out} \times C_{in} \times K_h \times K_w.

An RGB layer therefore combines evidence across red, green, and blue. Later layers similarly combine all incoming feature channels unless grouped or depthwise convolution restricts those connections.

Control the Spatial Output

For one input dimension of size NN, kernel size KK, padding PP, stride SS, and dilation DD, the output size is

Nout=⌊N+2P−D(K−1)−1S⌋+1.N_{out} = \left\lfloor\frac{N + 2P - D(K-1) - 1}{S}\right\rfloor + 1.
Parameter What it controls Typical effect
Kernel size Local pattern extent Larger kernels use more context and parameters
Stride Distance between kernel positions A stride of 2 roughly halves spatial resolution
Padding Treatment of image borders “Same” padding can preserve resolution for stride 1
Dilation Spacing between kernel elements Expands coverage without adding weights

A dilation of 2 makes a 3 × 3 kernel sample across a 5 × 5 area. Padding prevents maps from shrinking too quickly, but padded border values are synthetic and can influence edge responses.

Stack Layers into a Hierarchy

One convolution sees only a local region. Stacking layers makes later activations depend on progressively larger input regions, while nonlinear activations between layers allow the network to represent more than one linear filter.

Early layers often respond to edges and color changes. Middle and deep layers combine those responses into textures, shapes, object parts, or other task-specific features.

Use Specialized Variants

  • 1 × 1 convolution mixes channels independently at each position and can change channel count.
  • Depthwise separable convolution performs one spatial filter per input channel, then uses a 1 × 1 convolution to mix channels. It can greatly reduce computation.
  • Transposed convolution applies the transpose of the convolution operator to increase spatial resolution. It is not a mathematical inverse or deconvolution.
  • Grouped convolution divides channels into groups, reducing connections and computation while preserving some channel mixing.

Why It Works for Vision

  • Locality: Nearby pixels and features often form meaningful visual patterns.
  • Weight sharing: Reusing a detector across locations drastically reduces the number of learned parameters.
  • Translation equivariance: Shifting the input tends to shift the feature map by the same amount, before boundary and stride effects.
  • Hierarchical composition: Repeated local operations build broad, complex features from simpler ones.
  • Useful inductive bias: The architecture encodes assumptions about images that can improve data efficiency compared with unrestricted dense connections.

Design Trade-offs

  • A local kernel cannot model distant interactions until depth, downsampling, or dilation gives it a sufficiently large receptive field.
  • Stride reduces computation and expands context quickly, but discards spatial detail and can introduce aliasing.
  • Dilation preserves resolution but may sample sparse grids and create gridding artifacts.
  • Standard convolution mixes every input and output channel, which can dominate computation in wide layers.
  • Translation equivariance does not imply invariance to rotation, scale, viewpoint, or deformation.
  • Transposed convolution can create checkerboard artifacts when kernel size and stride interact poorly.

Related