flâneur

Model alignment protects against accidental harms, not intentional ones

normaltech.ai · 1,883 words · saved by 1 readers

The hand wringing about failures of model alignment is misguided

By Arvind Narayanan, Sayash Kapoor, and Seth Lazar Preventing harms from AI is important. The AI safety community calls this the alignment problem. The vast majority of development effort to date has been on technical methods that modify models themselves. We’ll call this model alignment, as opposed to sociotechnical ways to mitigate harm. The main model alignment technique today is Reinforcement Learning with Human Feedback (RLHF), which has proven essential to the commercial success of chatbots. But RLHF has come to be seen as a catch-all solution to the dizzying variety of harms from…

related reading