NeuralDocs

Advanced Topics / Deep Reinforcement Learning

Deep Reinforcement Learning

Advanced

Combines neural networks with reinforcement learning so agents learn policies from rewards.

Core concepts

Agentpolicy π(a | s)Environmentaₜsₜ₊₁rₜ₊₁
At each step the agent observes state sₜ, chooses action aₜ from its policy π, and the environment returns reward rₜ₊₁ and next state sₜ₊₁. The goal is to maximise expected discounted return.
yt=rt+γ(1−dt)max⁡a′Qθˉ(st+1,a′)LDQN(θ)=ED[(yt−Qθ(st,at))2]\boldsymbol{\begin{aligned}y_t&=r_t+\gamma(1-d_t)\max_{a\prime}Q_{\bar\theta}(s_{t+1},a\prime)\\\mathcal{L}_{\mathrm{DQN}}(\theta)&=\mathbb{E}_{\mathcal{D}}\left[(y_t-Q_\theta(s_t,a_t))^2\right]\end{aligned}}

This is a DQN target and squared TD-error loss, not a direct neural-network assignment. Use 0 ≤ γ < 1 for discounted continuing tasks; γ = 1 requires additional conditions such as finite episodes. dₜ = 1 for a terminal transition (not merely a time-limit truncation), D is the replay buffer, and θ̄ denotes fixed target-network parameters. Do not backpropagate through yₜ.

  • Agent, environment, state, action, reward
  • Policy π(a|s)
  • Value function V(s) and action-value Q(s, a)

Algorithms

  • DQN — deep Q-learning with replay buffer
  • Policy gradients / REINFORCE
  • Actor-Critic, A3C, PPO
  • RLHF for aligning language models

Interview questions & answers

  1. Q1. How does reinforcement learning differ from supervised learning?

    Answer: An RL agent learns through interactions and rewards rather than a fixed set of correct output labels. Its actions influence future observations, and rewards may be delayed. Learning must therefore handle exploration and assign credit to actions across time.

  2. Q2. Why does DQN use experience replay and a target network?

    Answer: Replay samples stored transitions to reduce short-term correlation and reuse experience. A slowly updated target network makes bootstrapped targets more stable than using the constantly changing online network alone. True terminal transitions do not bootstrap, while time-limit truncations generally should.

  3. Q3. What is the difference between value-based and policy-based methods?

    Answer: Value-based methods learn expected returns, such as action values, and derive actions from them. Policy-based methods directly optimize a parameterized action-selection policy. Actor-critic methods combine a policy with a value estimate to guide learning; policy methods also naturally support continuous action distributions.

← Previous

Diffusion Models

Next →

Training Best Practices