Weights & Bias

1. Purpose & Motivation

1.1 Why weights and biases matter

  • They are the learnable parameters of a neural network
  • The entire goal of training is to find good values for weights and biases
  • They encode everything the model has learned from data

Without learnable parameters, a neural network would be a fixed, hand-coded function.

1.2 Role in computation

  • Weights control how much each input contributes to the output
  • Biases control the threshold at which a neuron activates
  • Together they define a linear transformation: z = Wx + b

2. Definition & Core Idea

2.1 Weights

A weight is a scalar value that multiplies an input signal. In a layer with n inputs and m outputs, the weight matrix W has shape (m, n).

2.2 Bias

A bias is a scalar added after the weighted sum. Each neuron has one bias term. The bias vector b has shape (m,).

2.3 Intuitive mental model

"Weights decide which inputs matter and by how much. Bias decides how easy it is for the neuron to fire."

3. The Linear Transformation

3.1 What weights and biases compute

z = W * x + b
  • W * x rotates and scales the input space
  • + b shifts (translates) the result

3.2 Geometric interpretation

Operation Geometric effect
W * x Rotation, scaling, shear
+ b Translation
f(z) Non-linear warping

Without bias, all hyperplanes must pass through the origin, severely limiting expressiveness.

4. How Weights and Biases Are Updated

4.1 Gradient descent

  1. Compute predictions (forward pass)
  2. Measure error with a loss function
  3. Compute gradients via backpropagation
  4. Update parameters: w = w - lr * dL/dw

4.2 Update rule

W_new = W_old - learning_rate * gradient_W
b_new = b_old - learning_rate * gradient_b

4.3 Why both need gradients

  • Weight gradients tell us how to adjust input importance
  • Bias gradients tell us how to shift activation thresholds
  • Both are computed simultaneously during backpropagation

5. Weight Initialization

5.1 Why initialization matters

  • Poor initialization causes vanishing or exploding gradients
  • Symmetry breaking: if all weights are identical, all neurons learn the same thing
  • Good initialization enables faster, more stable training

5.2 Common strategies

Strategy Description Best for
Xavier/Glorot Variance based on fan-in + fan-out Sigmoid, tanh
He (Kaiming) Variance based on fan-in ReLU variants
Zero init All zeros (for biases only) Bias terms
Random normal Small random values Simple cases

Key insight: Weights are initialized randomly; biases are typically initialized to zero.

6. Weight Properties in Practice

6.1 Parameter count

  • A dense layer with 512 inputs and 256 outputs has 512 * 256 + 256 = 131,328 parameters
  • Weight matrices dominate the parameter budget
  • Modern networks have millions to billions of parameters

6.2 Weight sharing

  • CNNs share weights across spatial locations (kernels)
  • This dramatically reduces parameter count
  • Weight sharing encodes the assumption that patterns are location-invariant

7. Common Issues

7.1 Problems related to weights

Problem Cause Solution
Vanishing gradients Weights shrink gradients Better init, skip connections
Exploding gradients Weights amplify gradients Gradient clipping, normalization
Dead neurons Weights push into zero region Leaky ReLU, careful init

8. One-Sentence Summary

Weights and biases are the learnable parameters that define what a neural network knows, with weights controlling input importance and biases shifting activation thresholds.

9. Related Knowledge Pages