← Back to all news
arXiv Computation and Language By Zheyuan Zhang, Zehao Wen, Alvin Zhang, Andrew Wang, Jianwen Xie, Daniel Khashabi, Tianmin Shu

AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents

Read the original on arXiv Computation and Language →

AgentOdyssey is a new evaluation framework that procedurally generates open-ended text games with rich entities, dynamic worlds, and long-horizon tasks to test test-time continual learning agents. The framework measures game progress and provides diagnostic tests on world knowledge acquisition, episodic memory, exploration, action diversity, and model cost. Experiments across six games show that even the best agents fall far short of human performance, and supervised fine-tuning leads to catastrophic forgetting, highlighting the need for better parametric continual learning methods.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 7

Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

Learn2Play Bench is a new benchmark that tests how well large language model agents learn from experience in unfamiliar, text‑based games with novel or counterintuitive rules. The benchmark provides reproducible feedback, automatic scoring, and varied game instances to evaluate learning across repeated attempts and transfer to new situations. Findings show that retaining full action records aids learning, human players outperform agents, and the choice of harness significantly impacts performance and inference cost.

By Yibo Li, Jinhang Qiu, Zhi Zheng, Qianyun Guo, Jiaying Wu, Shuo Ji, Bryan Hooi
llmsagentsbenchmarks
More like this →
arXiv Computation and Language
2d ago

Can Large Language Model Agents Learn by Watching Tutorial Videos?

The paper investigates whether large language model (LLM) agents can acquire new knowledge from tutorial videos without updating their parameters. It introduces Tutorial Agentic Learning (TAL), a non‑parametric continual‑learning framework that builds a structured tutorial‑book memory via a two‑stage adversarial process, and retrieves information through a Table‑of‑Contents tree during inference. On the VidLearn multimodal QA benchmark—comprising 72 chess tutorial videos—TAL outperforms baseline models, achieving up to 113% relative gains on training data and 91% on test data, and improves in real‑world chess openings in 69% of head‑to‑head games, with additional evidence of generality in a second domain.

By Haoyu Zhao, Zafeirios Fountas, Jun Wang
llmsragagentsmultimodalbenchmarkssafety
More like this →
arXiv AI
Jun 30

Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

arXiv:2507. 05257v4 Announce Type: replace-cross Abstract: Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component-memory, encompassing how agents memorize, update, and retrieve long-term information-is under-evaluated due to the lack of benchmarks.

By Yuanzhe Hu, Yu Wang, Julian McAuley
llmsragagentsbenchmarks
More like this →