NeuralDocs

Foundations / Regularization & Generalization

Regularization & Generalization

Intermediate

Techniques that prevent overfitting so the model performs well on unseen data.

Bias–variance trade-off

Underfitting (high bias) means the model is too simple; overfitting (high variance) means it memorises training data. Monitor training vs. validation loss to diagnose.

model complexityerrorsweet spot
Training errorValidation error
As model complexity grows, training error keeps falling while validation error falls then rises. Left of the sweet spot the model underfits (high bias); right of it, it overfits (high variance). Regularisation pushes you back toward the sweet spot.

Techniques

Ltotal=Ldata+λ∥w∥22\boldsymbol{\mathcal{L}_{\mathrm{total}}=\mathcal{L}_{\mathrm{data}}+\lambda\lVert\mathbf{w}\rVert_2^2}

λ ≥ 0 controls the L2 penalty, whose gradient is 2λw. This is an objective penalty; decoupled weight decay in AdamW is not generally equivalent to adding it to an adaptive optimizer’s loss.

  • L2 weight decay / L1 penalties.
  • Dropout — randomly zero activations during training.
  • Batch Normalization and Layer Normalization.
  • Data augmentation — flips, crops, noise, mixup.
  • Early stopping on validation loss.
  • Label smoothing.

Interview questions & answers

  1. Q1. How can training and validation curves reveal overfitting?

    Answer: Overfitting is suggested when training loss continues to improve while validation loss worsens or the gap between them grows. Consider data augmentation, a smaller model, stronger regularization or early stopping, while checking that the validation set represents the intended deployment data.

  2. Q2. How does dropout behave during training and inference?

    Answer: During training, dropout randomly zeros activations. Inverted dropout also scales retained activations to preserve their expected value. During ordinary inference, dropout is disabled and all activations are used; inference should use evaluation mode.

  3. Q3. How does early stopping prevent overfitting?

    Answer: Early stopping monitors a validation metric and stops training after it fails to improve for a chosen patience period. Restore the best validation checkpoint rather than automatically taking the last one. Keep the test set out of checkpoint selection.

← Previous

Optimizers

Next →

Feedforward Networks (MLP)