A simple technical explanation of RLH(AI)F | Kairos.fm

Understanding reinforcement learning from human, or AI, feedback.

Large language models (LLM) like ChatGPT or Claude are trained on a huge amount of text. This training has made LLMs really good at predicting the next word so that it says coherent things. For example, if you start with “once upon a,” the model predicts “time” as the next word, having seen this pattern many times during its training. Researchers then built on the LLM’s next-word prediction skills, training LLMs to perform tasks like answering questions or summarising text. However, training on unfiltered internet text means these models can also: Help people with harmful tasks, ranging…

saved by

• Alex Yun
• Jackson Kozlowski

related reading

• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• [2309.00267] RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackarxiv.org
• RLHF & Post-Training Course by Nathan Lambertrlhfbook.com
• LLM Training: RLHF and Its Alternativesmagazine.sebastianraschka.com
• Introduction | RLHF and Post-Training Book by Nathan Lambertrlhfbook.com
• rlhfbook.com/book.pdfrlhfbook.com
• GenAI Handbookgenai-handbook.github.io
• [2402.14740] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsarxiv.org
• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• Reinforcement learning from human feedback - Wikipediaen.wikipedia.org
• 2307.12950.pdfarxiv.org
• GitHub - ash80/RLHF_in_notebooks: RLHF (Supervised fine-tuning, reward model, and PPO) step-by-step in 3 Jupyter notebooksgithub.com