Bhagyesh Kumar
3 followers · 26 following · 858 views
on the atlas — 270
- Neural Networks Are Already Clean37 savers
- OneWM — Can World and Action Modeling Benefit Each Other under Unified Causal Training?1 savers
- Andy Liu on X: "LLM developers train their models to follow prosocial values, but post-training can influence model behavior in unexpected ways. In “Predicting Alignment Generalization with Value Representations”, we ask whether downstream effects of value alignment can be predicted in advance. https://t.co/BifkS92piy" / X1 savers
- [1606.03137] Cooperative Inverse Reinforcement Learning3 savers
- Quantifying Truesight With SAEs · Gwern.net4 savers
- Guardian Angels: LLM Personalization for Productivity and Security · Gwern.net30 savers
- I am retiring from fulltime writing (& pseudonymity) to launch Guardian Angel | Hacker News1 savers
- Geodesic Pretraining Agenda: H2 2026 - Google Docs1 savers
- A Comedy of Estimators: On KL Regularization in RL Training of LLMs | alphaXiv1 savers
- ResearchTrails1 savers
- Scientific discovery as meta-optimization: a combinatorial optimization case study | alphaXiv1 savers
- Design_of_experiments2 savers
- [2610.06778] IdeaLens: Detecting AI Ideas in Long-form Writing1 savers
- Value Learning — AI Alignment Forum1 savers
- Projects - Textbook from the Future2 savers
- “Alignment Engineering” vs. “Misalignment Science” — LessWrong9 savers
- You can’t imitation-learn how to continual-learn — LessWrong6 savers
- Steering Language Model Goals with Value Transplant | alphaXiv1 savers
- Open Problems in Constitutional Preference Reconstruction | alphaXiv1 savers
- Formality is Favored: Unraveling the Learning Preferences of Large Language Models on Data with Conflicting Knowledge | alphaXiv1 savers
- Decoding Human Preferences in Alignment: An Improved Approach to Inverse Constitutional AI | alphaXiv1 savers
- NYCC - a Hugging Face Space by LAISR1 savers
- Steven Byrnes on X: "When people in AI alignment discuss “VNM rationality”, “utility maximizers”, etc., they’re generally talking about agents that care exclusively about the state of the world in the distant future. And then there’s a separate discourse about whether we should expect future powerful" / X1 savers
- [2609.38645] Alignment via Training Against Probes Without Losing Monitorability1 savers
- When Text and Numbers Disagree: Evidence Arbitration in Large Language Models | alphaXiv1 savers
- Whose Facts Win? LLM Source Preferences under Knowledge Conflicts | alphaXiv1 savers
- Character training can mitigate reward hacking, but can also make it harder to detect — LessWrong2 savers
- Believe It or Not: How Deeply do LLMs Believe Implanted Facts? | alphaXiv1 savers
- Inverse Constitutional AI: Compressing Preferences into Principles | alphaXiv1 savers
- Collective Constitutional AI: Aligning a Language Model with Public Input | alphaXiv1 savers
- Model Spec Midtraining: Improving How Alignment Training Generalizes | alphaXiv1 savers
- Stress-Testing Model Specs Reveals Character Differences among Language Models | alphaXiv1 savers
- LLM Bidders Preserve the Mechanism-Level Orderings of Human Bidders | alphaXiv1 savers
- From Belief Entrenchment to Robust Reasoning in LLM Agents - ACL Anthology1 savers
- How Value Induction Reshapes LLM Behaviour | alphaXiv1 savers
- Verbalizing Subliminal Learning Effects Using Text Optimization | alphaXiv1 savers
- Explaining Patterns in Data with Language Models via Interpretable Autoprompting | alphaXiv1 savers
- What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data | alphaXiv1 savers
- VibeCheck: Discover and Quantify Qualitative Differences in Large Language Models | alphaXiv1 savers
- Tell me about yourself: LLMs are aware of their learned behaviors | alphaXiv1 savers
- Stick to your Role! Stability of Personal Values Expressed in Large Language Models | alphaXiv1 savers
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs | alphaXiv1 savers
- Aaryam Sharma on X: "1/ When you run speculative decoding, when is a draft token actually accepted? For standard speculative sampling, the answer is elegant: with target distribution p and draft distribution q (assuming context is implicit), the acceptance probability is: 1 − TV(p, q) However, for https://t.co/gT9ZQWlAsl" / X1 savers
- [2606.30265] When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding1 savers
- [2604.17596] Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories1 savers
- [2609.19101] Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations1 savers
- Hack-Verifiable Terminal Bench1 savers
- Our Value | MetaConscious Group1 savers
- The AI Singularity by Buck Shlegeris and Dani Murach - YouTube2 savers
- Towards safety cases for frontier AI training | OpenAI2 savers
- On the Value of Doing a PhD in the Age of AI12 savers
- The behavioral selection model for predicting AI motivations — AI Alignment Forum1 savers
- Towards Universal Post-Training for Robotics — Perry Dong6 savers
- Dominique Paul on X: "This was an unusually good week for robot learning blog posts 1/ @perryadong and @chelseabfinn: robotics needs a standard post-training recipe, the way LLMs got one https://t.co/GFZpxOP01x 2/ @bfl_ai released Flux 3 Action w/ a post on it - finally, another strong open-weights" / X1 savers
- [2401.10899] Concrete Problems in AI Safety, Revisited1 savers
- Problems | Notion1 savers
- MATS Reading List – Luke Drago4 savers
- [2609.21892] Real quadratic fields and finite quantum dilogarithms I1 savers
- [2609.20412] Stress-testing Alignment Midtraining1 savers
- [2609.20979] An elementary proof of the Komlós conjecture1 savers
- Coherence arguments imply a force for goal-directed behavior — LessWrong1 savers
- the woman of irori | Project Lawful1 savers
- AI Futures: Forecasting & Strategy | Lens Academy1 savers
- About This Course | Course Overview1 savers
- [2511.16035] Liars' Bench: Evaluating Lie Detectors for Language Models1 savers
- Evaluating honesty and lie detection techniques on a diverse suite of dishonest models1 savers
- AIEssays.pdf1 savers
- [2511.22662] Difficulties with Evaluating a Deception Detector for AIs1 savers
- The Talker Does Not Control The Doer (in Current AIs) — LessWrong2 savers
- Explaining Knightianism on one foot — LessWrong2 savers
- Bayesianism / Problem of the Priors - Essay - Google Docs1 savers
- Why I’m Joining Thinking Machines — Neil Chowdhury5 savers
- A beginning for mathematics – Proofs and Prompts1 savers
- Why are AI agents lying, cheating and coordinating? | Yoshua Bengio3 savers
- Research acceleration: The view inside OpenAI | OpenAI14 savers
- Slowly, then all at once - by Jasmine Li - The J-Space5 savers
- Stop trying to try and try32 savers
- Can We Survive Technology?6 savers
- Model Hermeneutics Agenda - Google Docs1 savers
- Counterarguments to the basic AI risk case - by Katja Grace1 savers
- Epoch Capabilities Index | Epoch AI3 savers
- [2503.14499] Measuring AI Ability to Complete Long Software Tasks2 savers
- evhub's Shortform — LessWrong2 savers
- [2503.14499] Measuring AI Ability to Complete Long Software Tasks1 savers
- [2606.30116] Open Problems in Constitutional Preference Reconstruction1 savers
- In Praise of Boredom - LessWrong2 savers
- Value is Fragile — LessWrong3 savers
- AI is an abundance of choice not a 1D spectrum1 savers
- Doom as a bad method not a utopia trade-off — LessWrong3 savers
- [math/9404236] On proof and progress in mathematics3 savers
- Pacing the Frontier4 savers
- Dario Amodei — We Must Pace the Frontier36 savers
- Gro-Tsen on X: "I see more than a slight inconsistency between the AI tech bros saying on the one hand: ‣ “We'll soon have AIs to do everything: don't worry about jobs being lost: this is all good because humans can just sit back and enjoy the good life.” — and on the other, …" / X1 savers
- Navier-Stokes Announcement - Clay Mathematics Institute1 savers
- OpenAI agents carried out an undisclosed cyber-attack on RubyGems1 savers
- Terence Tao on AI — a living summary — Terence Tao2 savers
- Using group theory to explore the space of positional encodings for attention5 savers
- AI 202755 savers
- Unstable Ontology – by Jessica Taylor4 savers
- Inverting qualia with group theory — LessWrong1 savers
highlights — 29
Reasons for patience
The Haste Consideration, Revisiteda better idea of the magnitude of cooperation gains would help us figure out how much to prioritize ECL.
Everett branches, inter-light cone trade and other alien matters: Appendix to “An ECL explainer” — EA ForumThis means that one is limited to only acausally cooperating with other agents who take acausal influence seriously.
Everett branches, inter-light cone trade and other alien matters: Appendix to “An ECL explainer” — EA ForumThe most important reason for our view is that we are optimistic about the following:
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrong"Chi doesn't have great ideas." With that out of the way, here are some of my thoughts:
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWronglet it fully solve decision theory using its own superior intellect
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrongAI will be philosophically competent
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrongECL increases the case for working on AI systems that benefit other value systems by 1.5x–10x.
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrongpotential downsides to working on making AI engage in ECL
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrongExamples of cooperative actions: making AGIs be cooperative (whether they be aligned or misaligned, and insofar as we build AGI at all); taking a more pluralistic stance towards morality than we otherwise would.
Evidential Cooperation in Large Worlds: Potential Objections & FAQ — LessWrongexpected utility maximization
Decision Theory FAQ — LessWrongpotential fragility of acausal influence
Cooperating with aliens and AGIs: An ECL explainer — LessWrongreinforcement learning with formal verification in-the-loop
The Scalable Formal Oversight Research Program — LessWrongconstrained decoding or constrained generation
The Scalable Formal Oversight Research Program — LessWrongAgents with formal methods
The Scalable Formal Oversight Research Program — LessWrongJocelyn Qiaochu Chen
The Scalable Formal Oversight Research Program — LessWrongAI safety is extremely important regardless of your AI timelines
The Scalable Formal Oversight Research Program — LessWrongare super good at flagging unsafe code,
The Scalable Formal Oversight Research Program — LessWrongreliability is independent of the problem
The Scalable Formal Oversight Research Program — LessWrongSFO is about the box, not the monster you put in it.
The Scalable Formal Oversight Research Program — LessWrongguarantees for tasks totally unrelated to codegen
The Scalable Formal Oversight Research Program — LessWrongwhat actuator the ASI is using
The Scalable Formal Oversight Research Program — LessWrongASI might hack your verifier
The Scalable Formal Oversight Research Program — LessWrongformal verification offers a clear direction for how one might implement audits
The Scalable Formal Oversight Research Program — LessWrongat variants of this basic idea without giving it a name.
The Scalable Formal Oversight Research Program — LessWrongbiggest problem is that specification is hard
How to Solve Secure Program Synthesis — LessWrongissue is referred to as elicitation.
How to Solve Secure Program Synthesis — LessWrongfour (increasingly specific) types of deceptive AIs: • Alignment fakers: AIs pretending to be more aligned than they are.1 • Training gamers: AIs that understand the process being used to train them (I’ll call this understanding “situational awareness”), and that are optimizing for what I call "reward on the episode" (and that will often have incentives to fake alignment, if doing so would lead to reward).2 • Power-motivated instrumental training-gamers (or “schemers”): AIs that are training-gaming specifically in order to gain power for themselves or other AIs later.3 • Goal-guarding schemers…
Scheming AIs Will AIs fake alignment during training in order to get power?The overall taxonomy of model classes
Scheming AIs Will AIs fake alignment during training in order to get power?