Daniel Tan's Shortform — LessWrong
Comment by Thomas Kwa - Some other possible reasons: * Safety is simply more good than acceleration is bad * Working on safety has an internal policy impact, e.g. create evals that show a model is unsafe and block training or release * Working on safety has an external policy impact, e.g. study of CoT monitoring caused OpenAI and other labs to not train on CoT * Safety work now prevents imminent takeover, e.g. if models 1-2 generations in the future will be x-risky and alignment is easy * Safety work can prevent obvious misalignment, which makes it easier to observe subtle alignment problems. (This applies to fixing broken RL environments, but not in every type of safety work because removing obvious misalignment can also obscure subtle alignment problems) * Safety work now builds capacity for the safety work needed to actually align AI (possibly during slowdown) * They're doing safety research that ~only has long-term applications (like Leo Gao's ambitious mechinterp work) There are an ~equal number of reasons why working on safety can be bad, only some of which involve acceleration, so I have to think about like 15 considerations daily when doing project selection, times the enormous number of possible projects.