LLMs Know More Than What They Say - by Ruby Pai

... and how that provides winning evals

LLMs Know More Than What They Say ... and how that provides winning evals Ruby Pai Aug 15, 2024 15 1 Share Log10 provides the best starting point for evals with an effective balance of eval accuracy and sample efficiency. Our latent space approaches boost hallucination detection accuracy with tens of examples of human feedback (bottom), and work easily with new base models, surpassing performance of fine tuned models without the need to fine tune. For domain specific applications, our approach provides a sample efficient way to improve accuracy at the beginning of app development (bottom). See

related reading

• Patterns for Building LLM-based Systems & Productseugeneyan.com
• The bitter lesson of LLM evalsparsed.com
• Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• Successful language model evals - Jason Weijasonwei.net
• LLM Evaluation doesn't need to be complicatedphilschmid.de
• LLM evaluation: a beginner's guideevidentlyai.com
• LLM-as-a-judge: a complete guide to using LLMs for evaluationsevidentlyai.com
• Building an LLM evaluation framework: best practices | Datadogdatadoghq.com
• Discovering Language Model Behaviors with Model-Written Evaluations — LessWronglesswrong.com
• Hybrid Evaluation: Scaling human feedback with custom evaluation modelsarjunbansal.substack.com
• [2411.00640] Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluationsarxiv.org