Learn2Play Bench is a new benchmark that tests how well large language model agents learn from experience in unfamiliar, text‑based games with novel or counterintuitive rules. The benchmark provides reproducible feedback, automatic scoring, and varied game instances to evaluate learning across repeated attempts and transfer to new situations. Findings show that retaining full action records aids learning, human players outperform agents, and the choice of harness significantly impacts performance and inference cost.
By Yibo Li, Jinhang Qiu, Zhi Zheng, Qianyun Guo, Jiaying Wu, Shuo Ji, Bryan Hooi
arXiv:2606. 05684v1 Announce Type: new Abstract: A central challenge for language agents is utilizing past experience to adapt to dynamic test-time conditions.
By Yunxiang Zhang, Yiheng Li, Ali Payani, Lu Wang
arXiv:2606. 04815v1 Announce Type: cross Abstract: Lifelong learning is essential for Large Language Model (LLM) agents operating in dynamic, interactive environments.
By Bo Mao, Jie Zhou, Yutao Yang, Xin Li, Xian Wei, Qin Chen, Xingjiao Wu, Liang He
The paper investigates whether large language model (LLM) agents can acquire new knowledge from tutorial videos without updating their parameters. It introduces Tutorial Agentic Learning (TAL), a non‑parametric continual‑learning framework that builds a structured tutorial‑book memory via a two‑stage adversarial process, and retrieves information through a Table‑of‑Contents tree during inference. On the VidLearn multimodal QA benchmark—comprising 72 chess tutorial videos—TAL outperforms baseline models, achieving up to 113% relative gains on training data and 91% on test data, and improves in real‑world chess openings in 69% of head‑to‑head games, with additional evidence of generality in a second domain.
By Haoyu Zhao, Zafeirios Fountas, Jun Wang
arXiv:2610.07792v1 Announce Type: cross
Abstract: Large language model agents are increasingly deployed to perform complex tasks in real-world environments. However, the knowledge required for correc...
By Haizhong Zheng, Yizhuo Di, Ranajoy Sadhukhan, Shuowei Jin, Beidi Chen
arXiv:2507. 05257v4 Announce Type: replace-cross Abstract: Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component-memory, encompassing how agents memorize, update, and retrieve long-term information-is under-evaluated due to the lack of benchmarks.
By Yuanzhe Hu, Yu Wang, Julian McAuley
arXiv:2605.14211v4 Announce Type: replace
Abstract: Long-horizon visuomotor tasks remain a fundamental challenge in AI, as current methods rely on hand-engineered rewards or action-labeled demonstrat...
By Benjamin Schneider, Xavier Schneider, Victor Zhong, Sun Sun
arXiv:2606. 02461v1 Announce Type: new Abstract: Language agents spend substantial inference time solving individual tasks, yet the experience acquired in one episode is often underutilized in future episodes.
By Yiheng Shu, Bernal Jim\'enez Guti\'errez, Saisri Padmaja Jonnalagedda, Yuguang Yao, Huan Sun, Yu Su
arXiv:2606. 02461v2 Announce Type: replace Abstract: Language agents spend substantial inference time solving individual tasks, yet the experience acquired in one episode is often underutilized in future episodes.
By Yiheng Shu, Bernal Jim\'enez Guti\'errez, Saisri Padmaja Jonnalagedda, Yuguang Yao, Huan Sun, Yu Su
SpeedrunBench is a new benchmark that tests large language model agents on their ability to develop and refine strategies for completing nine different video games as quickly as possible. The benchmark requires agents to repeatedly improve, reflect, and reason over long action horizons to beat both themselves and other players. Experiments show that while agents can approach human records on simple platformers, they lag behind humans on longer, more complex games within realistic resource limits.
By Yoshinari Fujinuma, Keisuke Kamahori, Ryuto Koike, Abdelrahman Madkour, Varun Prashant Gangal, Monty Bichouna, Martyna Markiewicz, Shivani Jain, Duncan Curtis, Rebecca Qian, Anand Kannappan
arXiv:2609.38881v1 Announce Type: new
Abstract: Real-time strategy (RTS) games require agents to coordinate economic development, production and construction, base defense, unit organization, and att...
By Xinhe Tian, Xiaoyue Zhang, Ziyou Zhang, Jiacheng Li, Xiaoqiang Jin, Qianchuan Zhao, Gaochen Cui
The paper introduces the Continual Learning Exam (CLE), a benchmark designed to evaluate how large language model (LLM) agents improve over time by learning from experience. CLE consists of 120 human-authored tasks organized into five-level ladders, totaling 600 levels across 89 professional roles in 12 domains, where agents must pass each level to unlock the next. Experiments with 17 frontier LLM configurations show that providing diagnostic feedback and allowing one revision improves median completion by 5.3 percentage points, yet the best system only clears 47.7% of levels and completes all five levels in 23.3% of tasks, underscoring the challenge of sustained learning from experience.
By Haoyu Zhao, Sitian Tu, Yuxuan Huang, Huichi Zhou, Zhangyue Yin, Jiahao Huang, Jingyuan Xu, Tianmiao Shi, Jin Shi, Nianxi Zhang, Yangmiao Zhang, Zhuolang Chan, Huiyan Hu, Chenchao Sheng, Yaqing Luo, Yue Chen, Huaxiu Yao, Meng Fang, Jun Wang