Batch Norm & Dropout

Training deep neural networks is difficult because as parameters update, the distribution of each layer's inputs changes—a phenomenon called internal covariate shift. This can cause activations to grow unbounded, gradients to vanish or explode, and learning to stall.

Additionally, networks with large capacity tend to memorize the training data, failing to generalize to unseen examples (overfitting).

Batch Normalization

Batch Normalization stabilizes training by explicitly re-centering and re-scaling the activations of hidden layers during the forward pass. For each mini-batch, it calculates the mean and variance, normalizes the activations, and then applies a learned scale ($\gamma$) and shift ($\beta$). This keeps activations within a healthy range where the gradients of activation functions (like ReLU or Tanh) are most informative.

Dropout

Dropout acts as a regularizer. During training, it randomly zeroes out a fraction of neurons in a layer with a probability $p$ on each forward pass. This prevents neurons from co-adapting too strongly to specific features of the training data, forcing the network to learn a more robust, redundant representation. During inference (testing), all neurons are kept active, but their outputs are scaled by $1 - p$ to compensate.