The paper introduces RuVerBench, a benchmark with 2,458 instances for evaluating the reliability of Large Language Models acting as judges (LaaJ) in verifying rubric compliance within agentic scenarios such as deep research and agentic coding. It systematically meta‑evaluates frontier LLMs, revealing that even the most advanced models perform well yet still produce substantial noise. The study also examines how prompt design, batching, and majority voting affect verification accuracy, noting that weaker models are more prompt‑sensitive, batched verification trades accuracy for efficiency, and majority voting offers diminishing returns.
By Yangda Peng, Yunjia Qi, Haotian Xia, Guanzhong He, Xintong Shi, Richeng Xuan, Songyuanyi Lu, Yixian Liu, Zhichao Hu, Yuhong Liu, Hao Peng
arXiv:2608. 11215v1 Announce Type: new Abstract: Simulating societies of many large language model (LLM) agents is expensive, yet the questions asked of such simulations are usually macroscopic: phase behaviour, stylised facts, and scaling with the number of agents $N$, not the cognition of any single agent.
By Igor Itkin
arXiv:2602.03053v2 Announce Type: replace
Abstract: Multi-Agent Systems (MAS) built on Large Language Models (LLMs) often exhibit high variance in their reasoning trajectories. Process verification,...
By Vishal Venkataramani, Haizhou Shi, Zixuan Ke, Austin Xu, Xiaoxiao He, Yingbo Zhou, Semih Yavuz, Hao Wang, Shafiq Joty
The paper introduces Bayesian Dialectical Argumentation (BDA), a method for aggregating answers from multiple large language models (LLMs) in a council setting. BDA treats each LLM’s typed moves—proposals, challenges, and concessions—as evidence in a classical annotator model, estimating per-agent reliability even when some agents are persistently unreliable. By weighting evidence according to these inferred reliabilities, BDA produces calibrated posterior probabilities for candidate answers and can invert unreliable agents instead of merely outvoting them, achieving superior calibration and robustness on both binary and multi-class benchmarks without extra LLM calls.
By Ionel Eduard Stan, Paolo Napoletano
arXiv:2607. 03953v1 Announce Type: cross Abstract: This study independently replicates and extends the Natural Language Tools (NLT) framework of Johnson et al.
By Alexander Somma, Isabelle Plante, Fred Premji
arXiv:2609.01600v1 Announce Type: cross
Abstract: Dynamic agent harnesses let language models change the software that shapes their own execution. This flexibility brings a new reasoning burden: a lo...
By Damien Sileo, Dimitri Kachler
arXiv:2507.21790v3 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) are becoming widely used to support various workflows across different disciplines, yet their potential in discr...
By Georges Sfeir, Gabriel Nova, Stephane Hess, Sander van Cranenburgh
arXiv:2606. 06523v1 Announce Type: new Abstract: Equipping Large Language Models (LLMs) to execute reliable multi-step workflows has become a central challenge in artificial intelligence.
By Ruida Wang, Jerry Huang, Pengcheng Wang, Xuanqing Liu, Luyang Kong, Tong Zhang
arXiv:2607. 10059v1 Announce Type: new Abstract: Agent systems based on large language models (LLMs) are increasingly deployed for autonomous tasks, yet existing evaluations mostly focus on task success rather than whether agents know when to abstain.
By Xun Liu, Yi Evie Zhang, Vira Kasprova, Parisa Rabbani, Pardis Sadat Zahraei, Tianyu Zhang, Ali Ebrahimpour-Boroojeny, Varun Chandrasekaran
arXiv:2605. 28114v2 Announce Type: replace Abstract: Language-model agents are moving from single-user assistants into persistent networks that build trust and reputation with one another, and the same models increasingly control physically embodied robots as well as software.
By Messi H. J. Lee
arXiv:2603. 00829v2 Announce Type: replace-cross Abstract: Safe deployment of Large Language Model (LLM) agents in autonomous settings requires reliable oversight mechanisms.
By Simon Storf, Rich Barton-Cooper, James Peters-Gill, Marius Hobbhahn
arXiv:2603. 14465v2 Announce Type: replace Abstract: While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions.
By Shengda Fan, Xuyan Ye, Yupeng Huo, Zhi-Yuan Chen, Yiju Guo, Shenzhi Yang, Wenkai Yang, Shuqi Ye, Jingwen Chen, Haotian Chen, Xin Cong, Yankai Lin