arXiv AI
Aug 12
VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?
arXiv:2608. 10875v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly deployed as personal assistants.
VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?
Large language model (LLM) agents are increasingly deployed as personal assistants. Existing evaluations, however, mostly use short, self-contained requests in static environments.
Related stories
arXiv AI
2d ago
ReLiveGym: Evaluating Long-Lived Agents over Weeks of Replayed Reality
ReLiveGym is a diagnostic environment that evaluates long‑lived language‑model agents over weeks of chronologically replayed real‑world streams such as news, market data, and social media. The tasks vary in time sensitivity, reasoning depth, and recurrence, and the study tests eight base language models to see how model choice and harness design—especially action timing—affect performance. Continuous learning from hindsight feedback is also examined to address failure modes in these long‑term tasks.
arXiv AI
Jun 29
LiveClawBench: Benchmarking LLM Agents on Complex, Real-World Assistant Tasks
arXiv:2604. 13072v2 Announce Type: replace-cross Abstract: OpenClaw-style personal assistants extend LLM agents from isolated tool use to open-ended, stateful, and personalized software environments.
arXiv AI
Jul 24
DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers
arXiv:2607. 20531v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly deployed over Model Context Protocol (MCP) servers, yet the benchmarks used to evaluate them score the final answer or a fixed "ground-truth" list of tools, both of which are fragile once the underlying data is live and stateful.
arXiv AI
Aug 6
OneDayAgent: Towards a Long-Horizon Harness for Autonomous Agents
arXiv:2608. 05013v1 Announce Type: cross Abstract: LLM agents are increasingly applied to open-ended everyday requests that span work, study, and life.
arXiv AI
2d ago
Agents Are Systems, Not Models: Rethinking Agentic Evaluation
The paper argues that evaluating agents as fixed models is insufficient, proposing instead to treat them as configurable systems. Using a new benchmark of four scientific tasks, the authors analyze how five configuration aspects—task information, reasoning, self‑verification, time budget, and backbone model—affect performance, noting that about 54% of outcome variance arises from run‑to‑run differences even with the same settings. The study finds that providing more task information has the strongest impact, while interactions among settings (e.g., extra time only helps with adequate information or model capability) and the choice of verification tools significantly shape agent behavior.
arXiv AI
Jul 8
Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
arXiv:2510. 19771v4 Announce Type: replace Abstract: LLM-based agents are increasingly moving towards proactivity: rather than awaiting instruction, they exercise agency to anticipate user needs and solve them autonomously.
arXiv Machine Learning
Jun 16
MyPCBench: A Benchmark for Personally Intelligent Computer-Use Agents
arXiv:2606. 16748v1 Announce Type: new Abstract: Current benchmarks for computer-use agents evaluate models in impersonal environments.
arXiv AI
Jul 7
AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
arXiv:2607. 05174v1 Announce Type: new Abstract: Language agents, i.
arXiv AI
Sep 2
CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?
arXiv:2609.01600v1 Announce Type: cross Abstract: Dynamic agent harnesses let language models change the software that shapes their own execution. This flexibility brings a new reasoning burden: a lo...
arXiv AI
Jul 21
ClawBench: Can AI Agents Complete Everyday Online Tasks?
arXiv:2604. 08523v2 Announce Type: replace-cross Abstract: AI agents may be able to assist with emails and documents, but can they reliably complete everyday online workflows on real websites?
arXiv AI
Jun 2
COMAP: Co-Evolving World Models and Agent Policies for LLM Agents
arXiv:2606. 02372v1 Announce Type: new Abstract: Equipping language agents with world models enables them to anticipate environment dynamics and evaluate candidate actions before execution.