Anthropic’s “Hot Mess” paper overstates its case (and the blog post is worse) — LessWrong
Comment by Kaarel - hmm, like i think there's a reasonable sense of "coherence" such that it plausibly doesn't typically increase with capabilities. i think the survey respondents here are talking about something meaningful and i probably agree with most of their judgments about that thing. for example, with that notion of coherence, i probably agree with "Google (the company) is less coherent now than it was when it had <10 employees" (and this is so even though Google is more capable now than it was when it had 10 employees) this "coherence" is sth like "not being a hot mess" or "making internal tradeoffs efficiently" or "being well-orchestrated". in this sense, "incoherence" is getting at the following things: * to what extent are different parts of the guy out of sync with each other (like, as a % of how well they could be in sync)? * to what extent is the guy leaving value on the table compared to using the same parts differently? are there many opportunities for beneficial small rearrangements of parts? * how many arbitrage opportunities are there between the guy's activities/parts? * to what extent does it make sense to see all the parts/activities of the guy as working toward the same purpose? with this notion, i think there are many naturally-occurring cases of someone becoming more capable but less "coherent". e.g. maybe i read a textbook and surface-level-learn some new definitions and theorems and i can now solve the problems in the textbook, but the mathematical understanding i just gained is less integrated with the rest of my understanding than usual for me given that i've only surface-level-learned this stuff (and let's assume surface-level-learning this didn't let me integrate other existing stuff better) — like, maybe i mostly don't see how this theorem relates to other theorems, and wouldn't be able to easily recognize contexts in which it could be useful, and wouldn't be able to prove it, and it doesn't yet really make intuitive sense to m