NeuralDocs

Types of Neural Networks / Transformers

Transformers

Advanced

Attention-based architectures that process all tokens in parallel; the foundation of modern LLMs and vision models.

Scaled dot-product attention

Each token creates a Query, Key and Value. Attention weights decide how much each token attends to every other token.

Tokensembedding+ Positionpositional enc.LayerNormpre-normMulti-Headself-attention+ Residualx + Attn(x)LayerNormFeed-ForwardMLP per token+ Residual→ next block
A pre-norm Transformer block: token embeddings plus positional information pass through multi-head self-attention and a position-wise feed-forward network, each wrapped in a residual connection with layer normalisation.
Attention⁡(Q,K,V)=softmax⁡ ⁣(QKTdk+M)V\boldsymbol{\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\!\left(\frac{QK^{\mathsf T}}{\sqrt{d_k}}+M\right)V}

d_k is the query/key feature dimension. Softmax acts across keys in each query row. M is an additive mask: 0 for allowed positions and −∞ for blocked positions; use M = 0 for unmasked attention. Every query must have at least one allowed key.

Transformer block

  • Multi-head self-attention
  • Position-wise feed-forward network
  • Residual connections + Layer Normalization
  • Positional encodings (sinusoidal, learned, RoPE)

Families

TypeExampleTask
Encoder-onlyBERTUnderstanding / classification
Decoder-onlyGPTText generation
Encoder-decoderT5Translation, summarisation
VisionViTImage classification on patches

Interview questions & answers

  1. Q1. Why is dot-product attention scaled by the square root of the key dimension?

    Answer: Under common variance assumptions, query-key dot products grow in variance as the key dimension increases. Scaling counteracts this growth, reducing extremely peaked softmax distributions and helping maintain useful gradients.

  2. Q2. Why does a transformer need positional information?

    Answer: Unmasked self-attention alone is permutation equivariant and does not explicitly identify token order. Positional encodings or mechanisms such as rotary embeddings supply position-dependent information so the model can learn sequence order and relative relationships.

  3. Q3. How do encoder-only and decoder-only transformers differ?

    Answer: Encoder-only models usually let each token attend to all input positions and are commonly used for representation and understanding tasks. Autoregressive decoder-only models use causal masking to block future tokens. Training can process positions in parallel, but generation typically proceeds token by token.

← Previous

Generative Adversarial Networks

Next →

Graph Neural Networks