Feedback Loops With Language Models Drive In-Context Reward Hacking

This is experimental HTML to improve accessibility. We invite you to report rendering errors. Use Alt+Y to toggle on accessible reporting links and Alt+Shift+Y to toggle off. Learn more about this project and help improve conversions. Language models influence the external world: they query APIs that read and write to web pages, generate content that shapes human behavior, and run system commands as autonomous agents. These interactions form feedback loops: LLM outputs affect the world, which in turn affect subsequent LLM outputs. In this work, we show that feedback loops can cause in-context reward hacking (ICRH), where the LLM at test-time optimizes a (potentially implicit) objective but creates negative side effects in the process. For example, consider an LLM agent deployed to increase Twitter engagement; the LLM may retrieve its previous tweets into the context window and make them more controversial, increasing engagement but also toxicity. We identify and study two processes th

Feedback Loops With Language Models Drive In-Context Reward Hacking Feedback Loops With Language Models Drive In-Context Reward Hacking Alexander Pan Erik Jones Meena Jagadeesan Jacob Steinhardt Abstract Language models influence the external world: they query APIs that read and write to web pages, generate content that shapes human behavior, and run system commands as autonomous agents. These interactions form feedback loops: LLM outputs affect the world, which in turn affect subsequent LLM outputs. In this work, we show that feedback loops can cause in-context reward hacking (ICRH), where th

related reading

• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• A simple technical explanation of RLH(AI)F | Kairos.fmkairos.fm
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• Surfacing Pathological Behaviors in Language Models | Transluce AItransluce.org
• Training a Misaligned Reward Seekeralignment.anthropic.com
• Reward hacking is becoming more sophisticated and deliberate in frontier LLMs — LessWronglesswrong.com
• Simulated Users & Sad LLMs1a3orn.com
• [2305.18290] Direct Preference Optimization: Your Language Model is Secretly a Reward Modelarxiv.org
• Reward hacking behavior can generalize across tasks — AI Alignment Forumalignmentforum.org
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• 2401.10020.pdfarxiv.org