MLP

What an MLP Does

A Multi-Layer Perceptron (MLP) maps a fixed-size input vector to a prediction. It is a practical default for tabular data and a common building block inside larger networks, including Transformer feed-forward blocks and classification heads.

Unlike architectures designed around images or sequences, an MLP assumes no special structure in its inputs. Every value in one layer can influence every value in the next.

Core Idea

An MLP repeatedly applies an affine transformation and a nonlinear activation. Each layer turns the current representation into one that makes the final task easier.

For layer ll, the computation is

h(l)=ϕ(W(l)h(l−1)+b(l))\mathbf{h}^{(l)} = \phi\left(\mathbf{W}^{(l)}\mathbf{h}^{(l-1)} + \mathbf{b}^{(l)}\right)

where h(0)=x\mathbf{h}^{(0)}=\mathbf{x}, W\mathbf{W} and b\mathbf{b} are learned parameters, and ϕ\phi is an activation such as ReLU or GELU.

How It Works

1. Accept a Feature Vector

The input layer receives values such as measurements, encoded categories, or features produced by another model. It passes these values to the first learned layer without changing them.

2. Mix Features in Hidden Layers

A dense layer computes Wx+b\mathbf{W}\mathbf{x}+\mathbf{b}. Because every output unit has a weight for every input unit, it can combine evidence from all available features.

The activation function makes the transformation nonlinear. Without it, any number of stacked dense layers would collapse into a single linear transformation and could only learn linear decision boundaries.

3. Produce a Task-Specific Output

The final layer matches the task: one logit for binary classification, one value per class for multiclass classification, or one or more continuous values for regression. The loss function determines how those outputs are trained.

4. Learn with Backpropagation

Backpropagation calculates how each parameter contributed to the loss. An optimizer updates the weights and biases, gradually changing the intermediate representations and the final prediction.

Why It Works

  • Feature interaction: Dense connections let the model learn combinations that hand-designed linear models miss.
  • Nonlinear composition: Stacking nonlinear transformations can represent complex functions efficiently.
  • Universal approximation: A sufficiently wide MLP with one hidden layer can approximate any continuous function on a compact domain to arbitrary accuracy.
  • Reusable computation: The same simple block works as a standalone model or as a component inside CNNs and Transformers.

The universal approximation theorem describes representational capacity, not practical success. It does not specify how wide the network must be, make optimization easy, or guarantee generalization to unseen data. In practice, depth often represents useful functions more efficiently than extreme width.

When to Use It

An MLP is a strong baseline for modest fixed-size feature vectors, especially tabular or already-embedded data. It is also useful when the data has no meaningful spatial or temporal ordering.

For raw images, audio, or long sequences, an architecture with an appropriate inductive bias usually uses parameters and data more efficiently.

Limitations

  • A dense layer with nn inputs and mm outputs needs nm+mnm+m parameters, so large inputs become expensive quickly.
  • The architecture does not preserve spatial neighborhoods, translation structure, or sequence order.
  • Treating every feature connection independently provides less task-specific guidance than convolution or recurrence.
  • High-capacity MLPs can overfit small datasets without regularization and careful validation.

Related