Hamel’s Blog - Optimizing LLM latency
hamel.dev · 2,685 words · saved by 1 readers
An exploration of inference tools for open source LLMs focused on latency.
Summary Below is a summary of my findings: 🏁 mlc is the fastest. This is so fast that I’m skeptical and am now motivated to measure quality (if I have time). When checking the outputs manually, they didn’t seem that different than other approaches. ❤️ CTranslate2 is my favorite tool, which is among the fastest but is also the easiest to use. The documentation is the best out of all of the solutions I tried. Furthermore, I think that the ergonomics are excellent for the models that they support. Unlike vLLM, CTranslate doesn’t seem to support distributed inference just yet. 🛠️ vLLM is…
related reading
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- LLM Engineer's Almanac - Advisormodal.com
- Optimizing inference · Hugging Facehuggingface.co
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Efficient LLM inferencefinbarrtimbers.substack.com
- Two different tricks for fast LLM inferenceseangoedecke.com
- LLM Inference Handbookhandbook.modular.com
- vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention | vLLM Blogblog.vllm.ai
- Kimi-K2.5 Inference Benchmark - Luminalluminal.com
- How is LLaMa.cpp possible?finbarr.ca
- Wafer | Continual Inferencewafer.ai
- Best practices to accelerate inference for large-scale production workloadstogether.ai