NeuralDocs

Foundations / Gradient Descent & Backpropagation

Gradient Descent & Backpropagation

Intermediate

Backpropagation efficiently computes gradients of the loss with respect to every weight using the chain rule.

Gradient descent

Parameters move in the negative-gradient direction, scaled by the learning rate η. For a smooth loss and a nonzero gradient, a sufficiently small step decreases the loss; an excessively large step can increase it.

θL(θ)start θ₀minimumsteps: −η·∇L
Gradient descent on a one-parameter loss L(θ): start at θ₀, compute the gradient (slope), and step in the opposite direction by η·∇L. Repeating this walks downhill to the minimum. A small η takes many tiny steps; too large an η can overshoot and bounce out of the valley.
θt+1=θt−ηt∇θL(θt)\boldsymbol{\boldsymbol{\theta}_{t+1}=\boldsymbol{\theta}_{t}-\eta_t\nabla_{\boldsymbol{\theta}}\mathcal{L}(\boldsymbol{\theta}_{t})}

θ contains the learnable parameters, t is the optimization step, and ηₜ > 0 is the learning rate.

The chain rule

For a network computing L = l(f₃(f₂(f₁(x)))), the gradient flows backward layer by layer. Each layer only needs its local derivative and the gradient arriving from above.

∂L∂w1=∂L∂a3∂a3∂a2∂a2∂a1∂a1∂w1\boldsymbol{\frac{\partial\mathcal{L}}{\partial w_1}=\frac{\partial\mathcal{L}}{\partial a_3}\frac{\partial a_3}{\partial a_2}\frac{\partial a_2}{\partial a_1}\frac{\partial a_1}{\partial w_1}}

This illustrates a single scalar computation path with a₁ = f₁(x), a₂ = f₂(a₁), and a₃ = f₃(a₂). For vectors, backpropagation uses Jacobian-transpose–vector products and sums contributions from every path.

Training loop

for epoch in range(epochs):
    for x, y in loader:
        y_hat = model(x)          # forward pass
        loss = loss_fn(y_hat, y)  # compute loss
        optimizer.zero_grad()
        loss.backward()           # backpropagation
        optimizer.step()          # update weights

Vanishing & exploding gradients

  • Repeated multiplication of small derivatives shrinks gradients toward zero in deep networks.
  • Remedies: ReLU, careful initialisation (He / Xavier), batch/layer normalisation, residual connections, gradient clipping.

Interview questions & answers

  1. Q1. How are backpropagation and gradient descent different?

    Answer: Backpropagation computes derivatives of the loss with respect to parameters by applying the chain rule backward through the computation graph. Gradient descent uses those derivatives to update parameters. Computing a gradient does not itself change the weights.

  2. Q2. What happens if the learning rate is too large or too small?

    Answer: A rate that is too large can overshoot useful regions and make training oscillate or diverge. A very small rate can make progress unnecessarily slow. Tune it using validation performance and learning curves; a schedule can reduce the rate as training progresses.

  3. Q3. What causes vanishing and exploding gradients?

    Answer: Backpropagation multiplies local derivatives through layers or time steps. Repeated contraction can shrink gradients, while repeated amplification can make them explode. Suitable initialization, residual connections and gated recurrent units help gradient flow; gradient clipping limits exploding gradients.

← Previous

Loss Functions

Next →

Optimizers