NeuralDocs

Foundations / Activation Functions

Activation Functions

Beginner

Non-linear activations let networks model complex functions. Without them, stacked affine layers collapse to a single affine map.

Common activations

Sigmoid

xf(x)

Tanh

xf(x)

ReLU

xf(x)

Leaky ReLU (α = 0.1)

xf(x)

GELU

xf(x)

Softmax (2 classes)

xf(x)
Each curve plots the activation output f(x) against its input x. Sigmoid and tanh saturate for large |x|; ReLU and its variants stay linear for positive inputs. Softmax is shown for two classes as the probability of class 1 versus the logit difference x = z₁ − z₂.
FunctionFormulaRangeTypical use
Sigmoidσ(x)=11+e−x\boldsymbol{\sigma(x)=\frac{1}{1+e^{-x}}}(0, 1)Binary output
Tanhtanh⁡(x)=ex−e−xex+e−x\boldsymbol{\tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}}(−1, 1)RNN hidden states
ReLUmax⁡(0,x)\boldsymbol{\max(0,x)}[0, ∞)Default for hidden layers
Leaky ReLUmax⁡(αx,x),0<α<1\boldsymbol{\max(\alpha x,x),\quad 0<\alpha<1}(−∞, ∞)Avoid dead neurons
GELUxΦ(x)\boldsymbol{x\Phi(x)}[≈ −0.170, ∞)Transformers
Softmaxpi=exi−m∑j=1Cexj−m\boldsymbol{p_i=\frac{e^{x_i-m}}{\sum_{j=1}^{C}e^{x_j-m}}}(0, 1) for C ≥ 2; sums to 1Multi-class output

Choosing an activation

Φ is the standard normal cumulative distribution function. Exact GELU has an attained minimum of approximately −0.170. For softmax, m = maxⱼ xⱼ improves numerical stability without changing the result; each probability lies in (0, 1) for finite logits and C ≥ 2 (C = 1 gives probability 1).

  • Start with ReLU (or GELU for transformers) for hidden layers.
  • Sigmoid/tanh saturate and cause vanishing gradients in deep stacks.
  • Match the output activation to the task: sigmoid for binary, softmax for multi-class, none for regression.

Interview questions & answers

  1. Q1. Why do hidden layers need non-linear activation functions?

    Answer: Composing only affine transformations still gives a single affine transformation, regardless of depth. Non-linear activations allow a network to represent curved decision boundaries and more complex functions.

  2. Q2. What is the dying ReLU problem, and how can you address it?

    Answer: A ReLU neuron can output zero for all relevant inputs when its pre-activation stays negative, leaving it with zero local gradient. A smaller learning rate, appropriate initialization or an activation with a nonzero negative-side slope, such as Leaky ReLU, can help.

  3. Q3. When should you use sigmoid instead of softmax at the output?

    Answer: Use sigmoid for binary classification or independent multi-label predictions. Use softmax when predicting one class from mutually exclusive categories because it normalizes scores into probabilities that sum to one. In practice, numerically stable loss functions often take raw logits directly.

← Previous

Neurons & the Perceptron

Next →

Loss Functions