Reinforcement Learning as a fine-tuning paradigm | Ankesh Anand

Reinforcement Learning should be better seen as a “fine-tuning” paradigm that can add capabilities to general-purpose foundation models, rather than a paradigm that can bootstrap intelligence from scratch.

Reinforcement Learning (RL) should be better seen as a “fine-tuning” paradigm that can add capabilities to general-purpose pretrained models, rather than a paradigm that can bootstrap intelligence from scratch. Most contemporary reinforcement learning works involve training agents “tabula-rasa”, without relying on any sort of knowledge about the world. So when solving a task(s), the agent not only has to optimize the reward function at hand, but in the process also discover how to see, how physics works, what consequences its actions have, how language works, and so forth. This tends to work o

related reading

• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• The upcoming GPT-3 moment for RL | Mechanize, Inc.mechanize.work
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• The Era of Experience Paper.pdfstorage.googleapis.com
• Just Ask for Generalization | Eric Jangevjang.com
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• RL's Razor: Why Online Reinforcement Learning Forgets Lessarxiv.org
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Modelsarxiv.org
• Generalist - GEN-0 / Embodied Foundation Models That Scale with Physical Interactiongeneralistai.com
• rlhfbook.com/book.pdfrlhfbook.com
• The State of Reinforcement Learning for LLM Reasoningmagazine.sebastianraschka.com