Backpropagation Explained

📚 What is Backpropagation?

Backpropagation = Backward Propagation of Errors

It's the algorithm that tells a neural network how much each weight contributed to the error, so it can adjust and improve.

🎯 Think of it like Archery:

You shoot an arrow → miss the bullseye → the further you missed, the more you adjust your aim. Backpropagation is how the neural network does this mathematically, millions of times per second.

🔗 The Neural Network Structure

INPUT
Layer

X₁

X₂

X₃

→

HIDDEN
Layer 1

→

HIDDEN
Layer 2

→

OUTPUT
Layer

Ŷ

Each connection has a weight (the strength of the connection)

⚙️ How Backpropagation Works

📥

Forward Pass
(Input data)

→

📤

Prediction
(Output)

→

📊

Calculate
Error

→

🔄

Backpropagate
the Error

→

🔧

Adjust
Weights

1

Forward Pass

Input data flows through each layer, multiplied by weights, until we get an output (prediction).

Ŷ = σ(W × X + b)

2

Calculate Loss (Error)

Compare prediction with actual value. Use a loss function like Mean Squared Error.

Loss = (1/2) × (Ŷ - Y)²

3

Backward Pass (The Magic!)

Go backwards through the network. Calculate how much each weight contributed to the error using the Chain Rule of calculus.

4

Update Weights

Adjust each weight slightly in the direction that reduces the error. Use a learning rate to control how big the adjustments are.

W_new = W_old - η × ∂Loss/∂W

🔗 The Chain Rule — The Math Behind It

Backpropagation uses the chain rule to calculate how changes in one layer affect the final output.

If: y = f(g(x))

Then: dy/dx = (dy/dg) × (dg/dx)

In neural networks, we propagate the gradient (rate of change) backward through each layer:

∂Loss/∂W₁ = ∂Loss/∂Ŷ × ∂Ŷ/∂H × ∂H/∂W₁
This chain continues through all layers
Each weight gets its own gradient showing its contribution to the error

💡 Key Concepts

⚠️ Learning Rate (η)
How big each weight update step is. Too high = overshooting. Too low = too slow.

Gradient Descent: Following the slope downhill to find the minimum error
Stochastic Gradient Descent: Using random mini-batches instead of the whole dataset
Momentum: Adding "inertia" to avoid getting stuck in local minima
Vanishing Gradients: When gradients become too small in deep networks (solved by ReLU, residual connections)
Exploding Gradients: When gradients become too large (solved by gradient clipping)

📋 TL;DR Summary

Forward Pass: Make a prediction
Calculate Error: How wrong was it?
Backward Pass: Trace error back through network
Update Weights: Adjust to reduce error
Repeat: Millions of times until well-trained

🔬 Backpropagation