Hybrid Evaluation: Scaling human feedback with custom evaluation models

...how to really get model based evals to work for you

Human review is the most popular evaluation method. Source: Menlo VC From talking with enterprises deploying LLM apps we’ve learned that incorporating human-in-the-loop feedback remains an important factor in evaluating LLM app output. In many situations, such as in customer support it’s still critical for a human agent to review the output of the LLM before it’s presented to the end user. This manual review becomes a limiting factor as it slows down the time before the LLM generated response can be shown to the end user. In addition, the manual review increases the cost of deploying an LLM…

related reading

• Patterns for Building LLM-based Systems & Productseugeneyan.com
• Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
• The bitter lesson of LLM evalsparsed.com
• What We’ve Learned From A Year of Building with LLMs – Applied LLMsapplied-llms.org
• Demystifying evals for AI agents \ Anthropicanthropic.com
• LLM Evaluation doesn't need to be complicatedphilschmid.de
• Successful language model evals - Jason Weijasonwei.net
• RLHF: Reinforcement Learning from Human Feedbackhuyenchip.com
• LLMs Know More Than What They Say - by Ruby Paiarjunbansal.substack.com
• LLM evaluation: a beginner's guideevidentlyai.com
• LLM-as-a-judge: a complete guide to using LLMs for evaluationsevidentlyai.com
• Building an LLM evaluation framework: best practices | Datadogdatadoghq.com