flâneur

RL's Deadly Triad Meets Optimization | helen qu

helenqu.com · 2,988 words · saved by 1 readers

The unsavory combination of function approximation, off-policy learning, and bootstrapping, or the so-called “deadly triad” of reinforcement learning, has a status approaching that of folklore for its ability to induce unstable learning dynamics/divergence in even the simplest of systems. While likely a household name among RL enthusiasts, I’ve seen surprisingly little discussion on its origins or first principles. I made this post to shed light on the deadly triad from the perspective of traditional optimization theory, with the goal of demonstrating that the underlying principles are nothing more exotic than well-established convergence guarantees for dynamical systems. If the deadly triad is known to cause instability, why not avoid it altogether? While today’s RL post-training pipelines rarely combine all three elements at once, each piece still survives for good reason: function approximation for scale, off-policy updates for data efficiency, and bootstrapping for fast learning. W

RL's Deadly Triad Meets Optimization | helen qu \( \renewcommand{\vec}[1]{\mathbf{#1}} \newcommand{\avec}{\vec{a}} \newcommand{\bvec}{\vec{b}} \newcommand{\cvec}{\vec{c}} \newcommand{\dvec}{\vec{d}} \newcommand{\evec}{\vec{e}} \newcommand{\fvec}{\vec{f}} \newcommand{\gvec}{\vec{g}} \newcommand{\hvec}{\vec{h}} \newcommand{\ivec}{\vec{i}} \newcommand{\jvec}{\vec{j}} \newcommand{\kvec}{\vec{k}} \newcommand{\lvec}{\vec{l}} \newcommand{\mvec}{\vec{m}} \newcommand{\nvec}{\vec{n}} \newcommand{\ovec}{\vec{o}} \newcommand{\pvec}{\vec{p}} \newcommand{\qvec}{\vec{q}} \newcommand{\rvec}{\vec{r}} \newcomma

related reading