Foundations / Optimizers
Optimizers
IntermediateOptimizers decide how gradients are turned into weight updates.
Variants of gradient descent
- Batch GD — uses the whole dataset per step; stable but slow.
- Stochastic GD — one sample per step; noisy but fast.
- Mini-batch GD — the practical default (batch sizes 32–4096).
Popular optimizers
| Optimizer | Idea | Notes |
|---|---|---|
| SGD + Momentum | Accumulates velocity of past gradients | Strong for vision |
| RMSProp | Scales by running avg of squared gradients | Good for RNNs |
| Adam | Momentum + adaptive scaling | Default choice |
| AdamW | Adam with decoupled weight decay | Standard for transformers |
Learning-rate schedules
- Step decay
- Cosine annealing
- Linear warm-up followed by decay
- One-cycle policy
Interview questions & answers
Q1. How does momentum improve on plain SGD?
Answer: Momentum accumulates a running direction from past gradients. It can reduce oscillations across steep directions and accelerate progress along consistent directions, though it still requires a suitable learning rate and does not guarantee better generalization.
Q2. What is the difference between Adam and AdamW?
Answer: Adam uses moving averages of gradients and squared gradients, with bias correction, to adapt updates for each parameter. AdamW applies weight decay separately from the adaptive gradient update; adding an L2 penalty to Adam's loss is generally not equivalent to this decoupled decay.
Q3. Why use learning-rate warm-up and decay?
Answer: Warm-up gradually increases the learning rate to reduce unstable updates early in training. Decay lowers it later to allow smaller parameter adjustments. The schedule depends on the model and training budget, so compare schedules under the same validation protocol.