The Bitter Lesson Six Years On - by Lydia Nottingham

Five examples of the Bitter Lesson in practice

The Bitter Lesson only becomes legible in hindsight, when a single paper erases an entire decade of handcrafted work. This post is the first in a series. In future posts, I’ll give five counterexamples (for now) to the Bitter Lesson, think about which fields are Bitter-Lesson-resistant, and discuss where neurosymbolic approaches might most enduringly complement ML. Field: Information Retrieval Authors: Khattab & Zaharia (2020) Title: “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT.” In the beginning, there were bag-of-word information…

related reading

• Aman's AI Journal • Primers • Ilya Sutskever's Top 30aman.ai
• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• GenAI Handbookgenai-handbook.github.io
• A History of Large Language Modelsgregorygundersen.com
• Andrej Karpathy — AGI is still a decade awaydwarkesh.com
• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• On the Tradeoffs of SSMs and Transformers | Goomba Labgoombalab.github.io
• LLM Resourcesforrestbicker.com
• 1810.04805arxiv.org
• [2005.14165] Language Models are Few-Shot Learnersarxiv.org
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Transfer Learninglena-voita.github.io