Training Search Agents with GRPO — Jasper Lu

This post is a hands-on introduction to reinforcement learning through training a search agent with group-relative policy optimization (GRPO). Search is a fun place to learn RL because it has so many levers, and each of them visibly changes how the model searches. It is also a domain where, in my experience, a well-designed reward function can influence how a model searches more effectively than s

This post is a hands-on introduction to reinforcement learning through training a search agent with group-relative policy optimization (GRPO). Search is a fun place to learn RL because it has so many levers, and each of them visibly changes how the model searches. It is also a domain where, in my experience, a well-designed reward function can influence how a model searches more effectively than system prompt changes or harness engineering. Our approach is based loosely on the Harness-1 paper from Jiang et al. We use their training dataset and a much-simplified version of their search…

related reading

• Why GRPO is Important and How it Worksghost.oxen.ai
• A vision researcher’s guide to some RL stuff: PPO & GRPO - Yuge (Jimmy) Shiyugeten.github.io
• GRPO++: Tricks for Making RL Actually Workcameronrwolfe.substack.com
• State of RL for reasoning LLMs | A. Weersaweers.de
• Interactive Visualization of RL Algorithms for LLM Trainingzcy233035.github.io
• The Math Behind DeepSeek: A Deep Dive into Group Relative Policy Optimization (GRPO)medium.com
• Reward is not the optimization target — LessWronglesswrong.com
• karl.pdfdatabricks.com
• Lightweight Guide to understanding GRPO and RL principles - Musings of Muraligitlostmurali.com
• Bite: How Deepseek R1 was trainedphilschmid.de
• RLHF | John Lambertjohnwlambert.github.io
• Harsh Bhatt (@harshbhatt7585) on Xx.com