instruction tuning and autoregressive distribution shift — LessWrong

[Note: this began life as a "Quick Takes" comment, but it got pretty long, so I figured I might as well convert it to a regular post.] …

x instruction tuning and autoregressive distribution shift — LessWrong AI Frontpage 41 instruction tuning and autoregressive distribution shift by nostalgebraist 5th Sep 2024 6 min read 5 41 [Note: this began life as a "Quick Takes" comment, but it got pretty long, so I figured I might as well convert it to a regular post.] In LM training , every token provides new information about "the world beyond the LM" that can be used/"learned" in-context to better predict future tokens in the same window. But when text is produced by autoregressive sampling from the same LM, it is not informative in th

related reading

• Modifying LLM Beliefs with Synthetic Document Finetuningalignment.anthropic.com
• GenAI Handbookgenai-handbook.github.io
• Large Language Models Must Be Taught to Know What They Don't Knowarxiv.org
• [2203.02155] Training language models to follow instructions with human feedbackarxiv.org
• [2502.17424] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMsarxiv.org
• Self-Adapting Language Modelsarxiv.org
• Position: It's Time to Optimize for Self-Consistencytime-for-consistency.github.io
• Self-CTRL: Self-Consistency Training with Reinforcement Learningarxiv.org
• How do LLMs generalize when we do training that is intuitively compatible with two off-distribution behaviors? — LessWronglesswrong.com
• Large Language Diffusion Modelsarxiv.org
• [2402.05119] A Closer Look at the Limitations of Instruction Tuningarxiv.org
• Training language models to follow instructions with human feedback.pdfproceedings.neurips.cc