Efficient Model Serving, Part 1 (Overview) - YouTube
youtube.com · 182 words · saved by 1 readers
0 of 2 lessons complete Follow along using the transcript. 0 of 2 lessons complete Follow along using the transcript.
Efficient model serving (Columbia MLSS'26) Part 1 - Overview. Prefill versus decode, arithmetic intensity, and the roofline plot that tells you which of the two you are fighting. Part 2 - Hardware. The physics of the box: bandwidth ladders, the FP8 and FP4 number formats, why a single DRAM access costs about 500 multiplies, and why compute grows roughly 4x per hardware generation while memory bandwidth only doubles. Part 3 - Serving. Continuous batching, PagedAttention, prefix caching, RadixAttention, and splitting prefill off from decode onto separate hardware. Fun fact: your API bill is…
saved by
related reading
- Reiner Pope – The math behind how LLMs are trained and serveddwarkesh.com
- Transformer Inference Arithmetic | kipply's blogkipp.ly
- How To Scale Your Modeljax-ml.github.io
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- Making Deep Learning go Brrrr From First Principleshorace.io
- Efficient LLM inferencefinbarrtimbers.substack.com
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
- How is LLaMa.cpp possible?finbarr.ca
- All About Rooflines | How To Scale Your Modeljax-ml.github.io
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Compression and Intelligencegreene.sh
- Quantization from the ground upngrok.com