Faulty reward functions in the wild | OpenAI
Reinforcement learning algorithms can break in surprising, counterintuitive ways. In this post we’ll explore one failure mode, which is where you misspecify your reward function. At OpenAI, we’ve recently started using Universe (opens in a new window) , our software for measuring and training AI agents, to conduct new RL experiments. Sometimes these experiments illustrate some of the issues with RL as currently practiced. In the following example we’ll highlight what happens when a misspecified reward function encourages an RL agent to subvert its environment by prioritizing the acquisition of reward signals above other measures of success. Designing safe AI systems will require us to design algorithms that don’t attempt to do this, and will teach us to specify and shape goals in such a way they can’t be misinterpreted by our AI agents. One of the games we’ve been training on is CoastRunners (opens in a new window) . The goal of the game—as understood by most humans—is to finish the