Jun 30, 2026
1 min
Intrinsic motivation
work in progress Inverse Reinforcement Learning work in progress Offline RL work in progress Diffusion policies work in progress …
Topic
Deep RL
Posts under this topic.
Jun 30, 2026
12 min
Natural gradients
Learning stability The deep networks used as function approximators in the methods presented until now were all optimized (trained) using …
Jun 30, 2026
11 min
AlphaGo
Go is an ancient two-opponents board game, where each player successively places stones on a 19x19 grid. When a stone is surrounded by four …
Jun 30, 2026
8 min
World models, Dreamer
World models The core idea of world models (Ha and Schmidhuber 2018) is to explicitly separate the world model (what will happen next) from …
Jun 30, 2026
9 min
Planning (MPC, TDM)
Model-based learning is not only used to augment MF methods with imaginary rollouts. It can also be use directly for planning, as the model …
Jun 30, 2026
20 min
Off-policy Actor-Critic
On-policy vs. off-policy Actor-critic architectures are generally on-policy algorithms: the actions used to explore the environment must …
Jun 30, 2026
7 min
Model-based-augmented model-free RL (Dyna-Q, I2A)
Dyna-Q Once a model of the environment is learned, it is possible to augment MF algorithms with MB transitions. The MF algorithm (e.g. …
Jun 30, 2026
4 min
Model-based RL
Model-free vs. model-based RL Figure 1: Model-based RL uses planning to find the optimal action to perform, while model-free RL caches the …
Jun 30, 2026
12 min
Maximum Entropy RL (SAC)
All the methods seen sofar focus on finding a policy (or value functions) that maximizes the obtained return. This corresponds to the …
Jun 30, 2026
10 min
Actor-Critic with Experience Replay (ACER)
The natural gradient methods presented previously (TRPO, PPO) are stochastic actor-critic methods, therefore strictly on-policy. Off-policy …
Jun 30, 2026
14 min
Policy optimization (TRPO, PPO)
Trust Region Policy Optimization (TRPO) Principle Schulman et al. (2015) extended the idea of natural gradients to allow their use for …
Jun 30, 2026
15 min
Deep Deterministic Policy Gradient (DDPG)
So far, the actor produces a stochastic policy $\pi_\theta(s)$ assigning probabilities to each discrete action or necessitating sampling in …
Jun 30, 2026
14 min
Advantage Actor-Critic (A3C)
Actor-critic algorithms The policy gradient theorem provides an actor-critic arhictecture that allow to estimate the PG from single …
Jun 30, 2026
13 min
Policy Gradient methods
Approximating directly the Q-values in value-based methods (DQN) suffers from many problems when using deep neural networks: The Q-values …
Jun 30, 2026
21 min
DQN variants (Rainbow)
Double DQN In DQN, the experience replay memory and the target network were decisive in allowing the CNN to learn the tasks through RL. …
Jun 30, 2026
10 min
Deep Q-network (DQN)
Limitations of deep neural networks for function approximation The goal of value-based deep RL is to approximate the Q-value of each …
Jun 30, 2026
12 min
Deep learning
Deep RL uses deep neural networks as function approximators, allowing complex representations of the value of state-action pairs to be …
Jun 30, 2026
7 min
Function approximation
All the methods presented before are tabular methods, as one needs to store one value per state-action pair: either the Q-value of the …
Jun 30, 2026
17 min
Temporal Difference learning
Temporal difference The main drawback of Monte Carlo methods is that the task must be composed of finite episodes. Not only is it not always …
Jun 30, 2026
10 min
Monte Carlo methods
In dynamic programming, policy evaluation was done by explicitly solving the Bellman equations. In Monte Carlo (and temporal difference) …
Jun 30, 2026
23 min
Markov Decision Process
Markov Decision Process Definition Reinforcement Learning methods apply to problems where an agent interacts with an environment in discrete …
Jun 30, 2026
15 min
Sampling and Bandits
n-armed bandits The n-armed bandit (or multi-armed bandit) is the simplest form of learning by trial and error. Learning and action …
Jun 30, 2026
10 min
Introduction
Deep reinforcement learning (Deep RL) is the integration of deep learning methods, classically used in supervised or unsupervised learning …