Neuron

1. Purpose & Motivation

1.1 Why the neuron matters

  • It is the fundamental building block of every neural network
  • All complex architectures (CNNs, Transformers, etc.) are composed of neurons
  • Understanding a single neuron is the key to understanding deep learning

A neuron is to a neural network what a cell is to a living organism -- the smallest unit of computation.

1.2 Biological inspiration

  • Biological neurons receive signals through dendrites
  • Signals are combined in the cell body
  • If the combined signal exceeds a threshold, the neuron fires through the axon

The artificial neuron is a simplified mathematical abstraction of this biological process.

2. Definition & Core Idea

2.1 Formal definition

An artificial neuron is a computational unit that takes a vector of inputs, applies a weighted sum plus bias, and passes the result through a non-linear activation function to produce an output.

2.2 Mathematical model

The output of a neuron is computed as:

z = w1*x1 + w2*x2 + ... + wn*xn + b
a = f(z)

Where:

  • x = input values
  • w = weights (learnable parameters)
  • b = bias (learnable scalar)
  • f = activation function
  • z = pre-activation (linear combination)
  • a = activation (neuron output)

2.3 Compact notation

In vector form:

z = w^T * x + b
a = f(z)

This compact notation scales cleanly to layers and full networks.

3. Components of a Neuron

3.1 Inputs

  • Can be raw features (first layer) or outputs of previous neurons (hidden layers)
  • Each input has an associated weight

3.2 Weights

  • Control the importance of each input
  • Learned during training via gradient descent

3.3 Bias

  • Allows the neuron to shift its activation threshold
  • Without bias, the decision boundary must pass through the origin

3.4 Activation function

  • Introduces non-linearity
  • Common choices: ReLU, sigmoid, tanh
Component Role
Weights Scale input importance
Bias Shift activation threshold
Activation function Introduce non-linearity

4. What a Single Neuron Can Do

4.1 Linear separator

A single neuron with a threshold activation implements a linear classifier -- it can separate data that is linearly separable.

4.2 Limitations

  • Cannot solve XOR or other non-linearly separable problems
  • This limitation motivated the development of multi-layer networks

5. Role in Networks

5.1 Layers of neurons

  • Neurons are organized into layers (input, hidden, output)
  • Each neuron in a layer connects to neurons in the next layer
  • The collective behavior of many neurons enables complex function approximation

5.2 From neuron to network

Single neuron → Linear decision
One hidden layer → Simple non-linear boundaries
Deep network → Hierarchical feature learning

Key insight: Individual neurons are simple, but their composition creates powerful representations.

6. Historical Context

6.1 Key milestones

Year Milestone
1943 McCulloch-Pitts neuron model
1958 Rosenblatt's Perceptron
1969 Minsky & Papert expose limitations
1986 Backpropagation revives the field

7. One-Sentence Summary

A neuron computes a weighted sum of its inputs plus a bias, passes the result through an activation function, and serves as the atomic unit from which all neural networks are built.

8. Related Knowledge Pages