arXiv AI By Philip Drammeh

Multi-Agent LLM Orchestration Achieves Deterministic, High-Quality Decision Support for Incident Response

Read the original on arXiv AI →

The Flow has not summarised this story yet — read it at arXiv AI.

arXiv AI
Aug 28

DuMateBench: Evaluating Autonomous Agents in Complex Real-World Workflows

DuMateBench is a new benchmark for autonomous agents that uses real user sessions from a large production platform, preserving interaction history, configurations, and workspace state. It contains 200 tasks across 8 scenarios and 17 capability categories, many requiring coordination of multiple capabilities. The benchmark tests agents in Docker containers with real-world complexities—Insufficient, Unstable, and Noisy—and evaluates performance with a hybrid deterministic and LLM-as-Judge protocol, revealing significant gaps in task completion across various agent frameworks and LLMs.

By Zechun Niu, Yukun Zhao, Jiaxin Zhang, Xu Shen, Jinhua Si, Han Tian, Can Xu, Yunfan Song, Jiaxin Mao, Yansong Gao, Yuchen Li, Jianmin Wu, Lingyong Yan, Shuaiqiang Wang, Dawei Yin