LSTM & GRU

LSTM gates

These are standard LSTM equations without peephole connections. [h; x] denotes concatenation, σ is sigmoid, and ⊙ denotes element-wise multiplication. Gate values lie between 0 and 1. Initialize h₀ and c₀ with zeros or learned vectors; gating mitigates rather than eliminates vanishing gradients.

• Forget gate fₜ — what to discard from the cell state.
• Input gate iₜ — what new information to store.
• Output gate oₜ — what to expose as hidden state.

GRU

The GRU merges gates into an update gate and a reset gate, with no separate cell state. It has fewer parameters and often performs similarly to LSTM.

Applications

• Time-series forecasting
• Speech recognition
• Language modelling (pre-transformer)

Interview questions & answers

1. Q1. How does an LSTM mitigate vanishing gradients?

   Answer: Its cell state has an additive update controlled by forget and input gates, providing a path that can preserve information and gradients across time. This improves long-range learning compared with a basic RNN, but does not eliminate all vanishing-gradient problems.

2. Q2. What are the main differences between an LSTM and a GRU?

   Answer: An LSTM has separate cell and hidden states, with input, forget and output gates. A GRU uses a single hidden state with reset and update gates. GRUs usually have fewer parameters at the same hidden width, but neither architecture is universally better.

3. Q3. What do the forget, input and output gates control?

   Answer: The forget gate controls retention of the previous cell state. The input gate controls how much candidate information enters the cell. The output gate controls how much of the transformed cell state becomes the exposed hidden state. These gates act element-wise.