Gradient Hacking is extremely difficult.

Epistemic Status: Originally started out as a comment on this post but expanded enough to become its own post. My view has been formed by spending a reasonable amount of time trying and failing to construct toy gradient hackers by hand, but this could just reflect me being insufficiently creative...

Epistemic Status : Originally started out as a comment on this post but expanded enough to become its own post. My view has been formed by spending a reasonable amount of time trying and failing to construct toy gradient hackers by hand, but this could just reflect me being insufficiently creative or thoughtful rather than the intrinsic difficulty of the problem. There has been a lot of discussion recently about gradient hackers as a potentially important class of mesaoptimizers. The idea of gradient hackers is that they are some malign subnetwork that exists in a larger network that steer the

related reading

• Did Claude 3 Opus align itself via gradient hacking? — LessWronglesswrong.com
• The Little Book of Deep Learningfleuret.org
• Reward Hacking in Reinforcement Learning | Lil'Loglilianweng.github.io
• Why Momentum Really Worksdistill.pub
• [1609.04747] An overview of gradient descent optimization algorithmsarxiv.org
• Training a Misaligned Reward Seekeralignment.anthropic.com
• [1906.01820] Risks from Learned Optimization in Advanced Machine Learning Systemsarxiv.org
• Gradient descent - Wikipediaen.wikipedia.org
• A Visual Explanation of Gradient Descent Methods (Momentum, AdaGrad, RMSProp, Adam) | Towards Data Sciencetowardsdatascience.com
• An overview of gradient descent optimization algorithmsruder.io
• Hypothesis: gradient descent prefers general circuits — LessWronglesswrong.com
• Systematic Reward Hacking and Prime Sprintsprimeintellect.ai