On heterogeneous objectives. Defending AI systems in which different… | by Paul Christiano | AI Alignment
If you don’t know how to accomplish a goal safely using one AI, don’t imagine you can do it safely with 2 or more AIs […] If you actually understood the cognitive computations that output the right thing to do, you wouldn’t have to imagine them distributed across multiple imaginary people. As far as I can tell, he means this to apply to the kind of AI control research I’ve been doing (or maybe the kind that Eric Drexler has been doing, but the discussion will be the same either way). I think that this view is wrong, and I think this particular mistake is both important and (in one very tiny corner of the world) common. One way to build an AI system is to pick an objective, pick a class of models (or policies or etc.), and then search for a model which has a high objective value. This is a very general methodology; it’s the bread and butter of modern machine learning. This methodology gives us a nice way to think about the scalability of AI control proposals. Once we have our objective,