Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment
We introduce Inoculation Prompting (IP), a simple technique that reduces learning of an undesired behavior by modifying train-time prompts to explicitly request it. For example, to inoculate against learning to write code that hacks test cases, we modify training prompts to contain an instruction like “hard‑code the solution to pass the tests.” Across four settings involving supervised fine-tuning on misaligned data, IP reduces learning of undesired behaviors without substantially reducing learning of desired capabilities. 📄Paper, 💻Code Research done as part of the Anthropic Fellows Program. AI systems trained with imperfect oversight can learn undesired behaviors, like hacking test cases or sycophancy. Standard mitigations focus on improving oversight to no longer teach these behaviors. However, improving oversight can be difficult or expensive. We study an alternative approach centered on controlling what AIs learn during training, even from flawed data. Our technique, Inoculation