AgentTime is a benchmark that tests whether AI agents can manage their own runtime by working for a specified duration, predicting their wall‑clock time, and estimating elapsed time afterward. It includes 222 tasks from 18 sources across coding, computer use, agentic work, and automated research, with duration‑following experiments that append a single instruction specifying how long to work. Results show significant variability in accuracy, with some agents deviating from requested runtimes by up to 2.9×, and many agents stopping prematurely even after completing a task.
By Michael Ofengenden, Maksym Andriushchenko
StoreBench is a live‑commerce environment where an autonomous agent runs a mid‑size online apparel store on a production‑grade backend, facing continuous customer orders, supplier price changes, and market shocks. The agent uses the same 29 merchant tools as a human operator, with a windowed operation budget that ties simulated time to actions, ensuring that model latency does not affect the simulation. Seven large language models were evaluated across 11 scenarios of 30–45 days and a full simulated year, but none matched the scripted smart‑triage policy, and human experts outperformed all models.
By Daksh Raghuvanshi, Ved Vedere, Yifan Wang
arXiv:2609.00874v1 Announce Type: new
Abstract: As agentic systems become compound systems, increasingly important decisions move above task execution itself: when should a higher-level controller pr...
By Ayushi Chadha
arXiv:2606. 00198v1 Announce Type: cross Abstract: While agents are increasingly spending more resources, today agent cost is mostly measured only after execution.
By Yuxiang Lin, Zihan Wang, Mengyang Liu, Yuxuan Shan, Longju Bai, Junyao Zhang, Xing Jin, Boshan Chen, Jinyan Su, Xingyao Wang, Jiaxin Pei, Manling Li
arXiv:2608. 05519v1 Announce Type: new Abstract: Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic.
By Jie Wu, Ming Gong, Feixiang Cheng, Qinqin Zhao
The paper investigates how a higher‑level controller in a hierarchical latent reasoner decides whether to preserve or replace a lower‑level strategy. Experiments across different training seeds show that learned revision timing can produce varied policies, but none surpasses a fixed, forced timing policy on the same checkpoint. The study highlights that state‑dependent control can alter actions without improving task performance, and that timing decisions are order‑sensitive and largely captured by a strong fixed schedule.
By Ayushi Chadha