Generalized Transformers from Applicative Functors

Transformers are a machine-learning model at the foundation of many state-of-the-art systems in modern AI, originally proposed in [arXiv:1706.03762]. In this post, we are going to build a generalization of Transformer models that can operate on (almost) arbitrary structures such as functions, graphs, probability distributions, not just matrices and vectors.

Generalized Transformers from Applicative Functors 11 February 2025 Generalized Transformers from Applicative Functors by Tuomas Laakkonen Transformers are a machine-learning model at the foundation of many state-of-the-art systems in modern AI, originally proposed in [arXiv:1706.03762] . In this post, we are going to build a generalization of Transformer models that can operate on (almost) arbitrary structures such as functions, graphs, probability distributions, not just matrices and vectors. We will do this using the language of applicative functors , and indeed many of the constructions he

related reading

• A Mathematical Framework for Transformer Circuitstransformer-circuits.pub
• Thinking like Transformersrush.github.io
• Transformers from Scratche2eml.school
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• The Annotated Transformernlp.seas.harvard.edu
• transformer_attention.pdfarxiv.org
• Transformer Explainer: LLM Transformer Model Visually Explainedpoloclub.github.io
• Transformers from Scratchbrandonrohrer.com
• Transformer (deep learning) - Wikipediaen.wikipedia.org
• The Annotated Transformernlp.seas.harvard.edu
• The Illustrated Transformer – Jay Alammar – Visualizing machine learning one concept at a time.jalammar.github.io
• A Introduction to Transformersarxiv.org