Quasimetric RL (QRL)

In goal-reaching reinforcement learning (RL), the optimal value function has a particular geometry, called quasimetric structure (see also these works). This paper introduces Quasimetric Reinforcement Learning (QRL), a new RL method that utilizes quasimetric models to learn optimal value functions. Distinct from prior approaches, the QRL objective is specifically designed for quasimetrics, and provides strong theoretical recovery guarantees. Empirically, we conduct thorough analyses on a discretized MountainCar environment, identifying properties of QRL and its advantages over alternatives. On offline and online goal-reaching benchmarks, QRL also demonstrates improved sample efficiency and performance, across both state-based and image-based observations. ICML 2023. arXiv 2304.01203. Tongzhou Wang, Antonio Torralba, Phillip Isola, Amy Zhang. "Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning" International Conference on Machine Learning (ICML). 2023.

Quasimetric RL (QRL) Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning Tongzhou Wang MIT CSAIL Antonio Torralba MIT CSAIL Phillip Isola MIT CSAIL Amy Zhang UT Austin, Meta AI ICML 2023 PMLR Proceedings --> [arXiv] [Code] [Thread🧵] --> Overview of Quasimetric RL (QRL) Quasimetric Geometry + A Novel Objective (Push apart start state and goal while maintaining local distances) = Optimal Value $V^*$ AND High-Performing Goal-Reaching Agents Abstract In goal-reaching reinforcement learning (RL), the optimal value function has a particular geometry, called quasimetric stru

related reading

• A (Long) Peek into Reinforcement Learning | Lil'Loglilianweng.github.io
• Part 1: Key Concepts in RL - Spinning Up documentationspinningup.openai.com
• Q-learning is not yet scalableseohong.me
• State of RL for reasoning LLMs | A. Weersaweers.de
• The Promise of Hierarchical Reinforcement Learningthegradient.pub
• Key Papers in Deep RL - Spinning Up documentationspinningup.openai.com
• c21f4ce780c5c9d774f79841b81fdc6d-Paper.pdfproceedings.neurips.cc
• Contrastive Learning as Goal-Conditioned Reinforcement Learningarxiv.org
• ML Mentorship: Some Q/A about RLevjang.com
• [1710.10044] Distributional Reinforcement Learning with Quantile Regressionarxiv.org
• Accelerating Goal-Conditioned Reinforcement Learning Algorithms and Researcharxiv.org
• An Updated Introduction to Reinforcement Learning | Sri's Blogsrianumakonda.com