From Sampling to Reinforce
With the growing importance of LLM post-training, mastery of Reinforce has become arguably as central as techniques like attention. The algorithm is critical to producing models that reason, act, and discover. Despite being mathematically concise, Reinforce is tricky to master. It is not uncommon to read papers in the area that are confused or buggy. My personal theory for this is that random sampling can actually hide a bunch of issues. It makes it hard to just "code your way" through problems. The goal of this blog is to build up to Reinforce from scratch. It takes the non-standard approach of doing it without any random numbers. Obviously, this cannot scale, but it will let us build intuition for boring things like elementary variance reduction techniques. Code is available at srush/sampling-to-reinforce. The blog assumes no knowledge of RL or math. The text and comments are written by a person; the code is written by AI. This project is inspired by Haskell distributions, lea, and M