Feedforward Networks (MLP)

Structure

Every neuron in a layer connects to every neuron in the next layer. The Universal Approximation Theorem states that an MLP with one hidden layer can approximate any continuous function, given enough units.

Example in PyTorch

import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256), nn.ReLU(),
    nn.Linear(256, 128), nn.ReLU(),
    nn.Linear(128, 10)   # logits for 10 classes
)

Use cases

• Tabular data
• Classification heads on top of other networks
• Function approximation

Interview questions & answers

1. Q1. What makes an MLP a feedforward network?

   Answer: Information moves through a sequence of layers from input to output without recurrent state or cycles in the forward computation. Fully connected layers mix all input features, and non-linear activations between them provide expressive power.

2. Q2. What does the Universal Approximation Theorem guarantee, and what does it not?

   Answer: Under suitable activation assumptions, a sufficiently wide one-hidden-layer network can approximate continuous functions arbitrarily well on a compact domain. The theorem does not guarantee efficient width, successful training, good generalization or an easy way to find the required parameters.

3. Q3. How do you count the parameters in a fully connected layer?

   Answer: Multiply the number of input features by the number of output units, then add one bias per output unit if biases are enabled. A layer with 784 inputs and 256 outputs has 200,704 weights and 256 biases, totaling 200,960 trainable parameters.