How the RWKV language model works | The Good Minima

I go through and explain a minimal implementation of RWKV in detail.

In this post, I will explain the details of how RWKV generates text. For a high level overview of what RWKV is and what is so special about it, check out the other post about RWKV . To explain exactly how RWKV works, I think it is easiest to look at a simple implementation of it. The following ~100 line code (based on RWKV in 150 lines ) is a minimal implementation of a relatively small (430m parameter) RWKV model which generates text. Minimal RWKV code import numpy as np from torch import load as torch_load # Only for loading the model weights from tokenizers import Tokenizer layer_norm = lam

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• microgptkarpathy.github.io
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Unreasonable Effectiveness of Recurrent Neural Networkskarpathy.github.io
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• The Annotated Transformernlp.seas.harvard.edu
• ali (@waterloo_intern) on Xx.com
• interpreting GPT: the logit lens — LessWronglesswrong.com
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• How LLMs Actually Work | 0xkato0xkato.xyz
• Inside the Transformer: The Life of a Token - Aleksa Gordićaleksagordic.com