Home | Substack

“The line separating good and evil passes right through every human heart."

“If a kingdom is divided against itself, it cannot stand. And if a house is divided against itself, it cannot be maintained.” - Mark 3:24 In my last post I argued that we should view intelligent agents as coalitions of cooperating and competing subagents. The crucial question is then: how can we characterize effectively-functioning coalitional agents? One standard criterion is coherence: the extent to which the agent acts as if it has consistent goals and beliefs. In other words, an agent is coherent insofar as disagreeing subagents are able to still act as a functional coalition. For…

related reading

• Do not conquer what you cannot defend — LessWronglesswrong.com
• Towards a scale-free theory of intelligent agency — AI Alignment Forumalignmentforum.org
• The hot mess theory of AI misalignment: More intelligent agents behave less coherently | Jascha’s blogsohl-dickstein.github.io
• Towards a scale-free theory of intelligent agencymindthefuture.info
• Patterns and problems in multiagent systemsanthropic.com
• ⿻ Symbiogenesis vs. Convergent Consequentialism — LessWronglesswrong.com
• On idea-driven ideasvitalik.eth.limo
• Power Lies Trembling: a three-book review — LessWronglesswrong.com
• [2608.03958] A game theory for foundation models shows new paths to rational cooperation through similarity inferencearxiv.org
• Subagents, akrasia, and coherence in humans — LessWronglesswrong.com
• Annoyingly Principled People, and what befalls them — LessWronglesswrong.com
• Of Swarms and Sand Godsblog.cosmos-institute.org