Open-world evaluations for measuring frontier AI capabilities

Introducing CRUX, a new project for evaluating AI on long, messy tasks

This post is 8,000 words long—it is our new collaborative paper on an emerging type of AI evaluation. The paper is also published in a PDF format here. Summary: AI models have started to saturate most major benchmarks. But does that mean they can build and ship a real product, or conduct a scientific experiment end-to-end, or navigate a government bureaucracy? Researchers have started testing AI in such real-world settings. We call these evals “open-world evaluations”. This essay defines open-world evaluations, surveys the lessons learned so far, and lays out best practices for conducting…

related reading

• Demystifying evals for AI agents \ Anthropicanthropic.com
• Measuring AI Ability to Complete Long Tasks - METRmetr.org
• Your Evals Will Break and You Won't See It Coming - Lun Wangwanglun1996.github.io
• Your AI Product Needs Evals – Hamel's Blog - Hamel Husainhamel.dev
• More compute, more capability: Why AI agent evaluations need to account for test-time compute | AISI Workaisi.gov.uk
• A Few Things I Learned About Evals - Ryan Bloomryanbloom.xyz
• Agent Evaluation: A Detailed Guidecameronrwolfe.substack.com
• Things I learned at OpenAI - by Karina Nguyen - sémaphoresemaphore.substack.com
• CAIS AI Dashboarddashboard.safe.ai
• Toward A Public Science of Model Behavior | Transluce AItransluce.org
• Frontier Risk Report (February to March 2026) - METRmetr.org
• AI agent evaluation frameworks for production - Vercelvercel.com