OS-Marathon is a new benchmark that tests computer‑use agents on vast‑horizon, repetitive tasks, covering 100 tasks across five scenarios and ten domains. The study shows that current state‑of‑the‑art agents perform poorly on these tasks, and that simply decomposing workflows into subtasks does not solve the problem. Introducing a cost‑friendly personalization method called GraphDemo, which adapts agents from a single human demonstration, improves performance, highlighting the value of human guidance for these challenging tasks.
By Jing Wu, Wenjie Ai, Daphne Barretto, Yiye Chen, Qingyu Chen, Yuhang He, Pranit Chawla, Nicholas Gyd\'e, Yanan Jian, Vibhav Vineet
AgentMercury is a scalable framework that synthesizes executable environments from high‑level business scenarios instead of task‑specific benchmarks. It creates a persistent world with entities, services, tools, and invariants, allowing diverse tasks and interaction trajectories to emerge naturally. The authors generated 4,783 environments across 14 industries and 50 countries, and training reinforcement‑learning agents on them improved performance on enterprise workflows and out‑of‑domain benchmarks, while the construction process itself can be learned to increase authoring success.
By Minbyul Jeong, Chanwoong Yoon
arXiv:2608. 15071v1 Announce Type: new Abstract: Learning from experience is critical for developing capable, self-improving large language model (LLM) agents.
By Tianxin Wei, Zhan Shi, Minhua Lin, Bing He, Zewen Liu, Yisi Sang, Yuanchen Bei, Xuying Ning, Jiaru Zou, Ting-Wei Li, Xiao Lin, Yanjun Zhao, Chi Wang, Benoit Dumoulin, Dakuo Wang, Jingrui He, Hanqing Lu
arXiv:2607. 01084v1 Announce Type: new Abstract: While Large Language Model (LLM) agents demonstrate proficiency in static benchmarks, their deployment in real-world scenarios is hindered by the dynamic nature of user queries, tool sets, and interaction dynamics.
By Song-Lin Lv, Weiming Wu, Rui Zhu, Zi-Jian Cheng, Lan-Zhe Guo
arXiv:2608. 05013v1 Announce Type: cross Abstract: LLM agents are increasingly applied to open-ended everyday requests that span work, study, and life.
By Jingsheng Zheng, Xinyuan Fang, Jintian Zhang, Zhengke Gui, Huajun Chen, Ningyu Zhang
The paper introduces Test-Time Environment Decomposition (TTED), a label‑free learning method that allows large language model agents to break down complex web environment observations into simpler sub‑modules during inference. By learning from experience within these sub‑environments, agents can compose the gained knowledge to improve performance in the full environment. Experiments on synthetic and realistic benchmarks show that this approach enhances compositional generalization and boosts real‑world web automation tasks.
By Junxuan Li, Zijun Liu, Ziyi Huang, Peng Li, Yuzhou Liu, Ming Yan, Yang Liu