Separating Prediction from Goal-Seeking — LessWrong
lesswrong.com · 957 words · saved by 1 readers
tl;dr: Mixing goal-directedness into cognitive processes that are working to truth-seek about possible futures tends to undermine both truth-seeking…
x Separating Prediction from Goal-Seeking — LessWrong Agency Group Rationality Negotiation Trust and Reputation Rationality Frontpage 34 Separating Prediction from Goal-Seeking by plex 19th Mar 2026 2 min read 0 34 tl;dr: Mixing goal-directedness into cognitive processes that are working to truth-seek about possible futures tends to undermine both truth-seeking and effective pursuit of your goals. Cleanly separating them has nice properties. [Epistemic Status: Mostly empirically observed rather than rigorously justified, but have seen it many times across different people and within myself so
related reading
- The behavioral selection model for predicting AI motivations — LessWronglesswrong.com
- Towards a scale-free theory of intelligent agencymindthefuture.info
- Towards a scale-free theory of intelligent agency — AI Alignment Forumalignmentforum.org
- Deep Deceptiveness — LessWronglesswrong.com
- Beliefs are Chosen to Serve Goals — LessWronglesswrong.com
- Goal Factoring — LessWronglesswrong.com
- New report: “Scheming AIs: Will AIs fake alignment during training in order to get power?”joecarlsmith.com
- Embedded Agency (full-text version) — LessWronglesswrong.com
- Why AIs aren't power-seeking yet — LessWronglesswrong.com
- Everything is Predictionblog.cassi-ai.com
- Fitness-Seekers: Generalizing the Reward-Seeking Threat Model — LessWronglesswrong.com
- From personas to intentions: towards a science of motivations for AI models — LessWronglesswrong.com