How many bits and joules for superintelligence?

I was thinking about two questions:

I was thinking about two questions: How much information does it take to train a superintelligence? How much energy does it take to run one? When we do RL on an LLM, we typically generate around 1,000-200,000 tokens of chain-of-thought to extract just O(1) bits of useful information, essentially just the reward signal: was this answer good or not? With supervised fine-tuning, by contrast, each token carries roughly 1 bit. The reason we use RL at all is that we don’t have good supervised data for the capabilities we care about. This becomes more visible with Evolutionary Strategies (ES),…

related reading

• How can LLM RL Work Despite Information-Theoretic Inefficiencyberen.io
• The Extreme Inefficiency of RL for Frontier Models - Toby Ordtobyord.com
• II. From AGI to Superintelligence: the Intelligence Explosion - SITUATIONAL AWARENESSsituational-awareness.ai
• The Artificial Intelligence Revolution: Part 1 - Wait But Whywaitbutwhy.com
• Ilya Sutskever — We're moving from the age of scaling to the age of researchdwarkesh.com
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• RL is even more information inefficient than you thoughtdwarkesh.com
• As Rocks May Think | Eric Jangevjang.com
• The upcoming GPT-3 moment for RL | Mechanize, Inc.mechanize.work
• Memory makes computation universal, remember?thinks.lol
• Compression and Intelligencegreene.sh
• Fermi estimate of future training runsdanieldewey.net