Finally, a Replacement for BERT

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

TL;DR This blog post introduces ModernBERT, a family of state-of-the-art encoder-only models representing improvements over older generation encoders across the board, with a 8192 sequence length, better downstream performance and much faster processing. ModernBERT is available as a slot-in replacement for any BERT-like models, with both a base (149M params) and large (395M params) model size. Click to see how to use these models with transformers ModernBERT will be included in v4.48.0 of transformers. Until then, it requires installing transformers from main: pip install…

related reading

• Getting Caught Up to Modern LLM Research | Samarth Goeldev.samarthgoel.com
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Illustrated BERT, ELMo, and co. (How NLP Cracked Transfer Learning) – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• Generalized Language Models | Lil'Loglilianweng.github.io
• What happened to BERT & T5? On Transformer Encoders, PrefixLM and Denoising Objectives - Yi Tayyitay.net
• 1810.04805arxiv.org
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Transformer Architectures · Hugging Facehuggingface.co
• The Annotated Transformernlp.seas.harvard.edu
• GenAI Handbookgenai-handbook.github.io
• The Bitter Lesson is coming for Tokenization – ⛰️ lucalplucalp.dev
• Transfer Learninglena-voita.github.io