Kernel

What a Kernel Does

A kernel is the small weight tensor that a convolutional layer applies to local regions. It converts each region into a response indicating how strongly that region expresses a learned pattern.

A layer learns many kernels in parallel. Their outputs become separate feature maps, giving the next layer multiple descriptions of the same spatial input.

Core Idea

Think of a kernel as a compact pattern template. Training adjusts its values so that its dot product is large or small for local structures that help the network solve its task.

For a two-dimensional convolution with CinC_{in} input channels, one standard kernel has shape

Cin×Kh×Kw.C_{in} \times K_h \times K_w.

One kernel produces one output channel. A layer with CoutC_{out} kernels stores CoutCinKhKwC_{out}C_{in}K_hK_w weights, plus an optional bias for each output channel.

How It Works

Match a Local Patch

At one position, the kernel multiplies its weights by the corresponding values in the input patch and sums the products. A bias may be added before the result passes through normalization or an activation function.

The kernel is then reused at other positions. Its repeated responses form a feature map that preserves where the detected evidence appeared.

Learn Useful Weights

Kernels are usually initialized with small random values, often using He/Kaiming initialization for ReLU networks. Backpropagation computes how each weight affected the loss, and an optimizer updates the weights.

The network is not told that a kernel must detect an edge or texture. Those roles emerge when they improve the training objective, and individual channels need not have a simple human-readable interpretation.

Build Features Across Depth

Layer depth Patterns often represented
Early Edges, color gradients, oriented textures
Middle Corners, curves, motifs, texture combinations
Deep Object parts and task-specific patterns

Early kernels act directly on pixels and can be displayed as small images. Deeper kernels operate across learned channels, so activation maximization or feature visualization is usually more informative than viewing their raw weights.

Grad-CAM answers a different question: it highlights input regions associated with a prediction rather than directly showing what a kernel's weights represent.

Choose the Kernel Size

Size Spatial coverage Weights per input channel Common role
1 × 1 One position 1 Channel mixing and dimensionality change
3 × 3 Small neighborhood 9 General-purpose spatial processing
5 × 5 Wider neighborhood 25 More context in one layer
7 × 7 Broad neighborhood 49 Stems or large-kernel architectures

Two stride-1 3 × 3 layers have a 5 × 5 theoretical receptive field. At equal channel width they use 18 weights per channel pair rather than 25, and an activation between them adds another nonlinear transformation.

That does not make 3 × 3 universally optimal. Large depthwise kernels, used in architectures such as ConvNeXt, can gather broad spatial context while avoiding the full cost of dense channel-to-channel convolution.

Compare Learned and Hand-Crafted Filters

Classical vision systems use fixed kernels such as Sobel for directional edges, Gabor filters for oriented frequency patterns, and the Laplacian for rapid intensity change.

CNN kernels are learned from data and can specialize to the dataset and objective. First-layer kernels often resemble classical edge or color filters, while deeper kernels compose features that would be difficult to design manually.

Why Kernels Matter

  • Compact feature detectors: A small set of weights can describe a useful local pattern.
  • Reusable evidence: Applying the same kernel everywhere allows one detector to work across spatial positions.
  • Diverse channels: Multiple kernels let a layer preserve different views of the same input.
  • Task adaptation: Learned weights discover features that support the actual objective rather than a fixed hand-designed pipeline.
  • Composability: Deeper kernels combine earlier feature maps into broader and more abstract representations.

Design Trade-offs

  • Larger kernels capture more context directly but increase parameters and computation in standard convolution.
  • Small kernels need more layers, dilation, or downsampling to model long-range structure.
  • Increasing the number of kernels raises representational capacity and the memory cost of activations.
  • Weight decay can limit weight growth, but excessive regularization may suppress useful specialization.
  • Depthwise kernels are efficient but do not mix channels without a separate pointwise or grouped operation.
  • Raw deep-layer weights are difficult to interpret, and a strong activation does not by itself explain a model's decision.

Related