SA-Bench is a diagnostic benchmark that evaluates how well large language model (LLM) agents reproduce scientific papers by measuring semantic alignment between generated code and the papers’ specifications. It covers 30 recent machine‑learning papers, decomposing each into 1,491 Semantic Alignment Units (SAUs) that assess numerical, methodological, protocol, and ordering fidelity. Across 12 generator configurations, the best model (Claude+PaperCoder) scores only 0.301 on average, highlighting that current LLMs often implement requirements incorrectly or with stubs.
By Xue Hu, Zewei Pan, Zeli Su, Zhou Liu, Wentao Zhang
PHMForge is an evaluation environment that tests large language model agents on industrial prognostics tasks using the Model Context Protocol (MCP). It provides 99 SME-authored scenarios across eight asset classes, 39 MCP-native tools wrapping published PHM algorithms, and deterministic evaluators that separate protocol fluency, reasoning, instrumentation, and tool use. The benchmark shows that the best agent configuration achieves 80.8% pass@1, with orchestration and tool sequencing errors as the main failure modes, and demonstrates the limitations of static retrieval for prognostic computation.
By Yusheng Li, Tianjun Feng, Yunfeng Chen, Chun-Yi Tsai, Yihan Sun, Ayan Das, Kaoutar El Maghraoui, Shuxin Lin, Dhaval Patel
AI agents are increasingly autonomous, posing significant risks that current designs hinder effective human oversight. The paper argues that oversight is degraded by both design choices and the cognitive decline of users who rely heavily on automation. It calls for prioritizing human cognitive needs in AI agent development, proposing design affordances and protocols to maintain critical judgment and counter skill atrophy.
By Margaret Mitchell, Avijit Ghosh, Samir Passi
STRIVE is a new framework for longitudinal radiology report generation that separates clinical reasoning into distinct Diagnosis, Attribute, and Temporal Change agents, each producing explicit evidence. The Temporal Change agent is refined with a Progression-Aware GRPO reward that differentiates direction-preserving errors from reversals. Verification occurs twice: a Consistency Gate aligns agent outputs before report generation, and a Validation Agent ensures the final report is supported by the aggregated evidence. On the Longitudinal-MIMIC dataset, STRIVE achieves superior clinical efficacy and more than doubles Longitudinal Change Concordance compared to the strongest baseline.
By Junyeong Maeng, Eunsong Kang, Heung-Il Suk
The paper investigates how adversarial signals can infiltrate large‑language‑model (LLM) based multi‑agent trading systems through the agents’ communication channels. By restricting the attacker to realistic inputs—source data and prompts—it studies role‑specific attacks on four functional roles (Analyst, Researcher, Trader, Risk Manager) and evaluates four communication topologies under data‑ and agent‑level attacks. Experiments across multiple assets, backbones, and target directions show that no architecture is inherently robust, highlighting the need for safer designs in agentic trading systems.
By CheolWon Na, Hao Ni, Lukasz Szpruch, Zhangyang Wang, Dhagash Mehta, Saurabh Nagrecha, Alejandro Lopez-Lira, Chanyeol Choi, Yongjae Lee, Jee-Hyong Lee
arXiv:2506.11578v5 Announce Type: replace
Abstract: Recently, large language models (LLMs) have demonstrated strong performance, ranging from simple to complex tasks. However, while large models achi...
By Byeongchan Lee, Jonghoon Lee, Dongyoung Kim, Jaehyung Kim, Kyungjoon Park, Dongjun Lee, Jinwoo Shin
MetaRAG introduces a belief-action aligned policy optimization framework for agentic retrieval-augmented generation (RAG). It incorporates Verify-first Action Generation and Internal Belief Probing to assess whether the current evidence is sufficient before taking an action, and uses a consistency reward gated by answer correctness to guide training. Experiments on seven public QA benchmarks demonstrate that MetaRAG improves the accuracy-efficiency trade-off over existing RL-based agentic RAG baselines, with benefits that transfer across research settings, optimizers, and model backbones.
By Qiuyi Qi, Tian Liang, Jiamu Wang, Jinjian Zhang, Wei Zhou, Pengcheng Zhu, Linjian Mo, Ming Kong, Jie Liu, Qiang Zhu
The paper investigates whether large language models (LLMs) correctly gauge their confidence when acting in a hidden‑information chess variant. In experiments where the location of a hidden royal piece is repeatedly relocated, the models’ stated probabilities about the piece’s position were almost never accurate at high confidence levels, with a calibration deficit concentrated in those high‑confidence events. Across multiple model configurations and providers, the same pattern emerged, and conventional evaluation metrics such as legality, cost, latency, and completion rate were found to be uncorrelated with belief quality, yet a model could still win the game despite poor confidence estimates.
By Bhushan Kashinath Joshi
The paper introduces TruthMarketTwin, a simulation framework that uses agent-based modeling to study large language model (LLM) agents in e‑commerce markets characterized by asymmetric information. It models bilateral trade where sellers and buyers make strategic decisions about listings, purchases, ratings, and recourse to maximize profit and utility. The study finds that LLM agents can autonomously exploit weaknesses in reputation‑based governance, but that warrant enforcement can reduce deception and alter strategic behavior.
By Shijun Lei, Quang Nguyen, Swapneel S Mehta, Zeping Li, Huichuan Fu, Xiaolong Zheng, Siki Chen, Yunji Liang, Philip Torr, Zhenfei Yin
EngiAI introduces a capability-based evaluation framework for tool-connected engineering agents, assessing workflow execution, retrieval-assisted parameter selection, HPC orchestration, and training-code authoring using execution traces and engineering artifacts. The framework was applied to four LLM backends on EngiBench Beams2D and Photonics2D, revealing that proprietary models outperform open-source ones in workflow completion and HPC orchestration, while indexed retrieval improves parameter selection. The study demonstrates that evaluating distinct skills separately provides clearer insight into failure mechanisms than end-to-end success rates alone.
By Gioele Molinari, Florian Felten, Soheyl Massoudi, Mark Fuge
The paper introduces a multi‑agent framework that lets large language models (LLMs) perform controlled experiments using scientific simulation models, specifically for pharmaceutical process design. Given a user query and baseline configuration, the system builds a structured task, designs and runs comparative simulations, interprets outcomes, and generates evidence‑based recommendations for optimizing process parameters. By integrating high‑fidelity simulations with LLMs, the approach yields more specific, actionable outputs and improves user‑rated correctness and helpfulness compared to language‑only reasoning.
By Yuchen Xia, Michael Weyrich, Nasser Jazdi, Johannes St\"umpfle, Johannes Sigel, Akshay Narla, Gavin K. Reynolds, Anna Jawor-Baczynska, Pol Llopart
EviDx is a new framework for evidence-aware active diagnosis that pairs patient-specific diagnostic environments with a clinical scaffold and an observer-guided runtime harness. The framework constructs interactive environments from raw clinical cases, organizes role-specialized agents and evidence tools, and regulates diagnostic termination by tracking uncertainty and evidence coverage. Experiments demonstrate that EviDx improves diagnostic performance and process stability while revealing model-dependent capability boundaries.
By Lihang Zeng, Shaoting Zhang, Xiaofan Zhang
PeakBench is a new benchmark designed to evaluate how large language model agents invoke multiple tools while respecting resource constraints and parallel execution. It provides executable multi‑tool workflows with dependency annotations and measured resource profiles, and introduces a two‑part evaluation framework that separates logical planning from physical scheduling. The study shows that strong logical planning alone does not guarantee safe or efficient execution, and that providing resource information can reduce overflows and improve utilization.
By Zhi-Kai Chen, Xu-Xiang Zhong, Song-Yan Li, De-Chuan Zhan, Han-Jia Ye
SPO++ is a new reinforcement learning method that eliminates the need for waiting on sibling rollouts by using a persistent prompt-level value estimate. It corrects a mismatch in the original SPO approach by standardizing terminal-outcome advantages under an action-token measure and reorganizes prompt evidence based on the policy event that generated it. Experiments on ALFWorld and Math‑TIR show that SPO++ improves online learning efficiency, with action-token-measure normalization identified as the key component.
By Kai Ruan, Jinghao Lin, Qianshan Wei, Ziqi Zhou, Zihe Huang
LUCAID is an agentic multimodal AI system designed for precision lung cancer pathology, integrating nine modules that cover the entire routine workflow—from quality control and tumor detection to histological subtyping, microenvironment profiling, cellularity quantification, and biomarker scoring (PD‑L1, MET, TROP‑2). The system generates automated structured reports and allows interactive querying of module outputs. In prospective clinical validation, LUCAID achieved 93.0% concordance with an expert‑panel reference standard for clinically actionable decisions, outperforming five experienced thoracic pathologists who ranged from 68.3% to 81.1% concordance.
By Marie-Lisa Eich, Kai Standvoss, Timo Milbich, Alexander M\"ollers, Miriam H\"agele, Philipp Anders, Lars Tharun, Hanna Kontradiuk, Sebastian Kons, Nader Aldoj, Recepcan Adig\"uzel, Adam Narai, Lukas H\"onig, Jonathan Striebel, Binru Yang, Mihnea P. Dragomir, Marvin Sextro, Philipp Keyl, Philipp Jurmeister, Rosemarie Krupar, Evelyn Ramberger, James Wells, Julika Ribbat-Idel, Andreas Kunft, Hussam Shuaib, Christian Groh\'e, Reinhard B\"uttner, David Horst, Klaus-Robert M\"uller, Lukas Ruff, Maximilian Alber, Frederick Klauschen, Simon Schallenberg
The paper reviews the evolution of object‑counting techniques from class‑specific density regression to open‑vocabulary, foundation‑model‑backed counters that can handle visual and textual prompts. It highlights that current evaluation relies on a few saturated benchmarks, leading to models exploiting statistical regularities rather than true generalization. The authors propose a five‑axis taxonomy and audit the literature across domains such as microscopy, remote sensing, crowd counting, and agriculture, identifying six structural contradictions and outlining a roadmap for robust, multimodal evaluation protocols.
By Joana Konadu Owusu, Shivanand Venkanna Sheshappanavar
The paper evaluates the use of large language models (LLMs) as judges for assessing conversational voice agents, comparing human judgments with GPT‑4.1 and GPT‑5 across telecom and retail interactions. It examines agreement, metric‑level correlations, and consistency across three evaluation configurations (p0, p1, p2) to determine how reliably LLMs can judge conversational quality and safety. The study finds that LLM‑based evaluation can be effective but its reliability varies by metric and configuration, suggesting a hybrid approach where LLMs handle scalable assessment while humans focus on metrics requiring contextual interpretation.
By Anupam Purwar, Shashank Singh, Kritika Srivastava
The paper introduces Evidence-First Reflection (EFR), a two-stage reflector for desktop GUI agents that separates visual difference extraction from outcome verification. EFR uses Set-of-Marks annotations to locate action sites and candidate changed regions, filters action-relevant changes, and then makes a final judgment based on cleaned evidence. Experiments on OSWorld-Verified and WindowsAgentArena show that EFR improves reflector accuracy by 7.11% and increases end-to-end task success by roughly 5–6%.
By Yijie Ma, Chaoyue Niu, Fan Wu, Guihai Chen
The paper introduces Pufibara, an agent harness designed to maintain engineering state and evidence across revisions in Modelica-based physical system modeling. It also presents a 232-task Modelica Agent Workflow Benchmark covering model repair, generation, and tuning, evaluated by an external benchmark-owned evaluator. Experiments show Pufibara outperforms Claude Code in task success and resource efficiency across two LLM backends.
By Zizhe Wang
The paper introduces rebuild‑dossier, an open‑source tool that locks an application’s real interface before code is written and enforces one‑test‑at‑a‑time building through automated checks. In experiments, a compliant agent failed a held‑back test while a rule‑breaking agent passed, showing that a passing test suite can be gamed. The study also demonstrates that the automated check mechanism, rather than interface‑locking alone, is crucial for reliable rebuilds, and that multi‑level verification catches errors that single‑level checks miss.
By Parker Fawcett