NeuralDocs

Types of Neural Networks / Recurrent Neural Networks

Recurrent Neural Networks

Intermediate

RNNs process sequences by maintaining a hidden state that carries information across time steps.

Recurrence

The same weights are reused at every time step. Training uses Backpropagation Through Time (BPTT).

RNNx₁y₁RNNx₂y₂RNNx₃y₃h₀h₁h₂h₃
An RNN unrolled through time: the same cell (shared weights) reads xₜ and the previous hidden state hₜ₋₁, then emits hₜ and an optional output yₜ.
ht=tanh⁡(Whht−1+Wxxt+bh)ot=Wyht+by\boldsymbol{\begin{aligned}\mathbf{h}_t&=\tanh(W_h\mathbf{h}_{t-1}+W_x\mathbf{x}_t+\mathbf{b}_h)\\\mathbf{o}_t&=W_y\mathbf{h}_t+\mathbf{b}_y\end{aligned}}

hₜ is the hidden state and oₜ is the output before any task-specific activation (for example, softmax for class probabilities). Bias vectors are included in both transformations. Initialize h₀ with zeros or a learned vector.

Sequence configurations

  • One-to-many — image captioning
  • Many-to-one — sentiment analysis
  • Many-to-many — translation, tagging
  • Bidirectional RNNs read the sequence in both directions

Limitations

  • Vanishing gradients make long-range dependencies hard to learn.
  • Sequential computation limits parallelism.

Interview questions & answers

  1. Q1. How does an RNN retain information from earlier time steps?

    Answer: At each step, the RNN computes a new hidden state from the current input and the previous hidden state using shared parameters. The state summarizes preceding context, but a basic RNN may struggle to preserve information over long sequences.

  2. Q2. What is backpropagation through time?

    Answer: BPTT unfolds the recurrent computation across time and applies backpropagation through that graph, accumulating gradients for the shared parameters. Truncated BPTT limits the number of steps traversed, reducing memory and compute but restricting gradient-based credit assignment.

  3. Q3. When is a bidirectional RNN unsuitable?

    Answer: A bidirectional RNN uses both earlier and later sequence elements to compute representations. It is unsuitable when future inputs are unavailable, such as strict real-time forecasting or causal next-token prediction. It can be useful when the entire input sequence is available.

← Previous

Convolutional Neural Networks

Next →

LSTM & GRU