NeuralDocs

Foundations / Neurons & the Perceptron

Neurons & the Perceptron

Beginner

The artificial neuron is the basic computational unit: a weighted sum of inputs passed through a non-linearity.

The artificial neuron

Each neuron receives inputs x, multiplies them by learnable weights w, adds a bias b and applies an activation function f.

x₁w₁x₂w₂x₃w₃Σ + bweighted sumfactivationŷ
One artificial neuron: each input xᵢ is scaled by its weight wᵢ, the weighted inputs are summed with a bias b, and the activation function f produces the output ŷ. A perceptron is this neuron with a step function as f.
y^=f ⁣(∑i=1dwixi+b)=f(wTx+b)\boldsymbol{\hat{y}=f\!\left(\sum_{i=1}^{d}w_i x_i+b\right)=f(\mathbf{w}^{\mathsf T}\mathbf{x}+b)}

Here d is the number of input features, w is the weight vector, b is the bias, and f is the activation function.

The perceptron (Rosenblatt, 1958)

The perceptron uses a step function as activation and can learn any linearly separable function. It famously cannot learn XOR, which motivated multi-layer networks.

y^=1[wTx+b≥0]w←w+η(y−y^)xb←b+η(y−y^)\boldsymbol{\begin{aligned}\hat{y}&=\mathbb{1}[\mathbf{w}^{\mathsf T}\mathbf{x}+b\geq 0]\\\mathbf{w}&\leftarrow\mathbf{w}+\eta(y-\hat{y})\mathbf{x}\\b&\leftarrow b+\eta(y-\hat{y})\end{aligned}}

For binary labels y ∈ {0, 1}, η > 0 is the learning rate. Update both weights and bias; the indicator is 1 when its condition holds and 0 otherwise.

Layers

  • Input layer — receives raw features.
  • Hidden layers — learn intermediate representations.
  • Output layer — produces predictions (logits, probabilities, values).
  • 'Deep' simply means many hidden layers.

Interview questions & answers

  1. Q1. What do the weights, bias and activation do in an artificial neuron?

    Answer: Weights control each input's contribution to a weighted sum. The bias shifts the decision boundary independently of the inputs. The activation transforms the result, usually introducing the non-linearity needed to model complex relationships.

  2. Q2. Why can a single perceptron not learn XOR?

    Answer: A single perceptron produces a linear decision boundary. XOR's positive and negative examples cannot be separated by one straight line, so it requires a model with non-linear features, such as a multi-layer network with non-linear activations.

  3. Q3. When does the perceptron learning algorithm converge?

    Answer: For a finite, linearly separable training set, the standard perceptron algorithm with a positive learning rate converges to a separating boundary after finitely many mistakes. For non-separable data, convergence to zero training error is not guaranteed.

← Previous

What is Deep Learning?

Next →

Activation Functions