← Back to all news
arXiv Computation and Language By Ruoyu Zhang, Zhaocen Liu, Weiyi Tian, Melissa Feng, Shilong Liu

OptionWorld: A Benchmark for Diagnosing Failures of LLMs as Long-Term Personal Assistants

Read the original on arXiv Computation and Language →

OptionWorld is a new benchmark that tracks how large language models handle long‑term personal assistant tasks, focusing on information gathering, decision making, actions, and outcomes over simulated daily life scenarios. The study finds that in 35.5% of failed cases the key facts are still present in the model’s last input, and identifies three main failure modes: missing or uncombined information, unused action opportunities, and unnecessary restrictions imposed by the agent. Follow‑up experiments show that providing missing facts, adding action reminders, and correcting erroneous restrictions can improve task success, highlighting gaps in state maintenance, deadline adherence, and assumption handling.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv AI
Oct 26

Knowing When to Stop: The Evidence-Action Gap in Coding Agents

The paper introduces the evidence-action gap, where coding agents persist in tasks even when completion is impossible, despite having sufficient evidence that outside help is needed. To quantify this, the authors present ImpasseBench, a benchmark that injects realistic blockers into repair tasks and marks the step at which evidence becomes sufficient. Across eight models and 19,371 trajectories, they find that many agents still exhaust their budget or make unsafe requests after the evidence threshold, but an escalation tool and external monitor can significantly reduce these failures.

By Tianyu Chen, Xinran Zhao, Vijay Viswanathan, Shoubin Yu, Mingyuan Zhou, Tongshuang Wu
agents
More like this →
arXiv AI
Oct 5

Sentry: Learning to Recover from LLM Agent Failures at Test Time

Sentry is a failure‑management layer for large language model agents that learns from failures at test time. It retrieves relevant lessons from an external playbook when a failure occurs, verifies recovery without task rewards, and stores new lessons only if recovery succeeds, keeping the playbook out of the agent’s context. Across multiple benchmarks, Sentry outperforms both runtime‑intervention and context‑evolution baselines, and its lessons transfer to unseen tasks.

By Changxiu Ji, Amy Lu, Qizheng Zhang, Kunle Olukotun
llmsagentsbenchmarks
More like this →