← Back to all news
arXiv AI By Detao Zeng, Guilin Ou, Nuo Chen, Yunhao Feng, Hongliang Wang, Qinli Yang, Junming Shao, Zhong Zhang

Async-RBench: Benchmarking Asynchronous Task Execution in LLM Agents

Read the original on arXiv AI →

Async-RBench is a new benchmark that tests how large language model agents handle asynchronous task execution. It includes 200 executable tasks across eight scenarios, with a scenario controller that delivers results based on task-state conditions and an Event Response Contract that defines required changes, preservation, avoidance, and re-verification. The benchmark introduces the Dynamic Replanning Score (DRS) to evaluate both the response process and outcome, and finds that agents generally perform worse with asynchronous delivery compared to batched delivery, highlighting challenges in reliable asynchronous task handling.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv AI
Aug 18

AstronOS: A Unified Execution Model and Runtime for Long-Horizon Agentic Systems

arXiv:2608. 16381v1 Announce Type: new Abstract: Agentic systems often organize execution and state around a single conversation, model invocation, or agent instance, even when real work spans many calls and stages.

By Zhenhang Nie (iFLYTEK Co., Ltd., Hefei, China), Gui Zheng (iFLYTEK Co., Ltd., Hefei, China), Xudong Sun (iFLYTEK Co., Ltd., Hefei, China), Tailong Zhu (iFLYTEK Co., Ltd., Hefei, China), Bin Zhang (iFLYTEK Co., Ltd., Hefei, China)
agentsbenchmarks
More like this →
arXiv AI
Jul 14

AgentAbstain: Do LLM Agents Know When Not to Act?

arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.

By Xun Liu, Yi Evie Zhang, Vira Kasprova, Parisa Rabbani, Pardis Sadat Zahraei, Tianyu Zhang, Ali Ebrahimpour-Boroojeny, Varun Chandrasekaran
llmsagentsbenchmarks
More like this →
arXiv AI
Sep 15

Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs

The paper introduces a new evaluation method called "same-input rerun" to assess the consistency of clinical language‑model agents across repeated runs. By replaying 1,000 MedAgentBench tasks with identical inputs, the authors find that action‑level outputs—such as test orders, medication requests, and referrals—vary significantly, even when benchmark scores remain unchanged. The study demonstrates that current benchmarks, which typically evaluate only a single run per task, can miss substantial behavioral divergence.

By Rohith Reddy Bellibatlu, Manpreet Singh, Zhoutian Han, Wenbin Zhang
llmsagentsbenchmarks
More like this →