NeuralDocs

Foundations / Loss Functions

Loss Functions

Beginner

A loss function measures how wrong the model is. Training is the process of minimising it.

Regression losses

MSE penalises large errors heavily; MAE and Huber loss are more robust to outliers.

error eloss
MSE e²MAE |e|Huber δ=1
Loss as a function of the error e = y − ŷ. MSE grows quadratically and punishes outliers heavily; MAE grows linearly; Huber (δ = 1) is quadratic near zero and linear beyond δ.
MSE=1n∑i=1n(yi−y^i)2MAE=1n∑i=1n∣yi−y^i∣\boldsymbol{\begin{aligned}\mathrm{MSE}&=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2\\\mathrm{MAE}&=\frac{1}{n}\sum_{i=1}^{n}|y_i-\hat{y}_i|\end{aligned}}

For n scalar predictions, yᵢ is the target and ŷᵢ is the prediction; both losses average over the examples.

Classification losses

LCE=−1n∑i=1n∑c=1Cyiclog⁡picLBCE=−1n∑i=1n[yilog⁡pi+(1−yi)log⁡(1−pi)]\boldsymbol{\begin{aligned}\mathcal{L}_{\mathrm{CE}}&=-\frac{1}{n}\sum_{i=1}^{n}\sum_{c=1}^{C}y_{ic}\log p_{ic}\\\mathcal{L}_{\mathrm{BCE}}&=-\frac{1}{n}\sum_{i=1}^{n}\left[y_i\log p_i+(1-y_i)\log(1-p_i)\right]\end{aligned}}

CE averages over n examples and C classes; yᵢc is a target distribution and pᵢc is the predicted class probability. BCE uses binary targets yᵢ ∈ {0, 1}. Logs are natural logarithms; use numerically stable losses on logits in practice.

  • Binary cross-entropy — paired with sigmoid output.
  • Categorical cross-entropy — paired with softmax output.
  • Focal loss — down-weights easy examples for class imbalance.

Other objectives

  • Contrastive / triplet loss — representation learning.
  • KL divergence — VAEs and knowledge distillation.
  • Adversarial loss — GANs.

Interview questions & answers

  1. Q1. How would you choose between MSE, MAE and Huber loss?

    Answer: MSE penalizes large errors quadratically and is sensitive to outliers. MAE penalizes errors linearly and is more robust, but is not differentiable at zero. Huber loss is quadratic near zero and linear for large errors, balancing smooth optimization with outlier robustness.

  2. Q2. Why is cross-entropy commonly used for classification?

    Answer: Cross-entropy penalizes low probability assigned to the target class and corresponds to negative log-likelihood under a categorical or Bernoulli model. Implementations that combine it with logits avoid numerical issues from separately computing probabilities and logarithms.

  3. Q3. How can you handle class imbalance in the training objective?

    Answer: Class-weighted losses increase the contribution of underrepresented classes, while focal loss reduces the influence of easy examples. Resampling is another option. Choose and validate the approach using task-relevant metrics such as precision, recall or PR-AUC rather than accuracy alone.

← Previous

Activation Functions

Next →

Gradient Descent & Backpropagation