ARIA RUN: A living benchmark for real-world AI
ARIA RUN (Reliability Under Naturalistic conditions) measures time-to-critical-mistake on real-world agentic tasks – including therapy, tutoring, caregiving, customer service – built on METR‘s time-horizon methodology, and adapted for the physical world.
Frontier AI has already revolutionized software. The physical world is next. Models are being deployed for therapy, tutoring, and caregiving, yet we have remarkably little insight into how effective or safe these systems are once they leave the lab. Existing benchmarks focus on accuracy on discrete tasks — did the model solve this problem, yes or no. Assistants deployed in the real world don’t work that way. They execute jobs: sequences of decisions unfolding over minutes, hours, or days, in which a single error can unwind everything that came before. The quality we measure is the same one reliability engineers have asked of aircraft, power plants, and surgical tools for decades: how long since the last mistake?