flâneur

AI leaderboards are no longer useful. It's time to switch to Pareto curves.

normaltech.ai · 3,833 words · saved by 1 readers

What spending $2,000 can tell us about evaluating AI agents

By Sayash Kapoor, Benedikt Stroebl, Arvind Narayanan Which is the most accurate AI system for generating code? Surprisingly, there isn’t currently a good way to answer questions like these. Based on HumanEval, a widely used benchmark for code generation, the most accurate publicly available system is LDB (short for LLM debugger).1 But there’s a catch. The most accurate generative AI systems, including LDB, tend to be agents,2 which repeatedly invoke language models like GPT-4. That means they can be orders of magnitude more costly to run than the models themselves (which are already pretty…

related reading