Backpropagation

Backpropagation is the algorithm that allows neural networks to learn. It calculates the gradient of the loss function with respect to every weight in the network.

When a neural network makes a prediction, the data flows forward from the inputs through the hidden layers to the output. This is the forward pass. Once the prediction is made, it is compared to the true target to calculate the error (or loss).

To reduce this error, we need to know how each weight contributed to it. This is where the backward pass comes in. Using the chain rule from calculus, the error signal is propagated backward from the output layer to the input layer. At each node, we calculate a "delta" (the local gradient), which tells us how much that node's output should change. We then multiply this delta by the input the weight received to find exactly how much to adjust the weight.

The Vanishing Gradient Problem

One common issue in deep networks is the vanishing gradient problem. When using activation functions like Sigmoid, large inputs cause the function to saturate (flatten out). In these flat regions, the derivative is very close to zero. When the backward pass multiplies by this near-zero derivative, the error signal vanishes, and the weights in earlier layers stop learning. Switching to ReLU helps alleviate this by maintaining a constant derivative of 1 for positive inputs.