Neuron
1. Purpose & Motivation
1.1 Why the neuron matters
- It is the fundamental building block of every neural network
- All complex architectures (CNNs, Transformers, etc.) are composed of neurons
- Understanding a single neuron is the key to understanding deep learning
A neuron is to a neural network what a cell is to a living organism -- the smallest unit of computation.
1.2 Biological inspiration
- Biological neurons receive signals through dendrites
- Signals are combined in the cell body
- If the combined signal exceeds a threshold, the neuron fires through the axon
The artificial neuron is a simplified mathematical abstraction of this biological process.
2. Definition & Core Idea
2.1 Formal definition
An artificial neuron is a computational unit that takes a vector of inputs, applies a weighted sum plus bias, and passes the result through a non-linear activation function to produce an output.
2.2 Mathematical model
The output of a neuron is computed as:
z = w1*x1 + w2*x2 + ... + wn*xn + b
a = f(z)
Where:
- x = input values
- w = weights (learnable parameters)
- b = bias (learnable scalar)
- f = activation function
- z = pre-activation (linear combination)
- a = activation (neuron output)
2.3 Compact notation
In vector form:
z = w^T * x + b
a = f(z)
This compact notation scales cleanly to layers and full networks.
3. Components of a Neuron
3.1 Inputs
- Can be raw features (first layer) or outputs of previous neurons (hidden layers)
- Each input has an associated weight
3.2 Weights
- Control the importance of each input
- Learned during training via gradient descent
3.3 Bias
- Allows the neuron to shift its activation threshold
- Without bias, the decision boundary must pass through the origin
3.4 Activation function
- Introduces non-linearity
- Common choices: ReLU, sigmoid, tanh
| Component | Role |
|---|---|
| Weights | Scale input importance |
| Bias | Shift activation threshold |
| Activation function | Introduce non-linearity |
4. What a Single Neuron Can Do
4.1 Linear separator
A single neuron with a threshold activation implements a linear classifier -- it can separate data that is linearly separable.
4.2 Limitations
- Cannot solve XOR or other non-linearly separable problems
- This limitation motivated the development of multi-layer networks
5. Role in Networks
5.1 Layers of neurons
- Neurons are organized into layers (input, hidden, output)
- Each neuron in a layer connects to neurons in the next layer
- The collective behavior of many neurons enables complex function approximation
5.2 From neuron to network
Single neuron → Linear decision
One hidden layer → Simple non-linear boundaries
Deep network → Hierarchical feature learning
Key insight: Individual neurons are simple, but their composition creates powerful representations.
6. Historical Context
6.1 Key milestones
| Year | Milestone |
|---|---|
| 1943 | McCulloch-Pitts neuron model |
| 1958 | Rosenblatt's Perceptron |
| 1969 | Minsky & Papert expose limitations |
| 1986 | Backpropagation revives the field |
7. One-Sentence Summary
A neuron computes a weighted sum of its inputs plus a bias, passes the result through an activation function, and serves as the atomic unit from which all neural networks are built.