The paper revisits the METR time‑horizon metric, which links human completion times of software tasks to the probability that an AI can solve them. Using splines and item‑response theory on 228 tasks and 26 AIs, the authors relax the linear‑log assumption and find that the relationship between human time and AI difficulty is nearly flat between 2–30 minutes but roughly linear outside that range. They provide improved time‑horizon estimates, better scoring performance, and diagnostic plots to assess construct validity, recommending that future benchmarks consider these diagnostics alongside the horizons.
arXiv:2503. 14499v4 Announce Type: replace Abstract: Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear.
By Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, Lawrence Chan
arXiv:2602. 07267v2 Announce Type: replace Abstract: Evaluating the real-world capabilities of AI systems requires grounding benchmark performance in human-interpretable measures of task difficulty.
By Fengyuan Liu, Jay Gala, Nilaksh, Dzmitry Bahdanau, Siva Reddy, Hugo Larochelle
arXiv:2606. 07157v1 Announce Type: new Abstract: Many efforts to ensure frontier AI models are safe rely on monitoring their chain-of-thought (CoT) reasoning.
By Dewi Gould, Francis Rhys Ward, Anders Cairns Woodruff, Rauno Arike, Josh Hills, Alex Serrano, Ida Caspary, Jason Ross Brown, Jo J. Jiao, Patrick Leask, Twm Stone, Ram Potham, Ionut Gabriel Stan, Harry Mayne, Simeon Hellsten, Shubhorup Biswas, Ariana Azarbal, William L. Anderson, Elle Najt, Ryan Greenblatt, Julian Stastny
arXiv:2502. 20502v2 Announce Type: replace Abstract: Recent advances in Artificial Intelligence (AI) have yielded powerful computational models that, by learning from vast amounts of human-generated data, are increasingly posited as approximate models of human cognition.
By Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L. Griffiths, Joshua B. Tenenbaum
AgentTime is a benchmark that tests whether AI agents can manage their own runtime by working for a specified duration, predicting their wall‑clock time, and estimating elapsed time afterward. It includes 222 tasks from 18 sources across coding, computer use, agentic work, and automated research, with duration‑following experiments that append a single instruction specifying how long to work. Results show significant variability in accuracy, with some agents deviating from requested runtimes by up to 2.9×, and many agents stopping prematurely even after completing a task.
By Michael Ofengenden, Maksym Andriushchenko