SWEPatchTest is a benchmark that evaluates software tests by measuring their ability to reproduce issues, accept correct patches, and reject incorrect ones across 619 tasks and 82,579 candidate patches. It introduces the Joint Validation Rate metric, revealing that even the best configurations only achieve 9.21% success when all three criteria are met. Using these validation scores to select demonstrations for supervised fine‑tuning can roughly triple the Joint Validation Rate and enhance downstream repair performance on multiple SWE‑bench datasets.
By Hua Lu, Pengcheng Wang, Han Li, Zichen Tian, Yanyu Chen, Huang He, Bingjin Chen
The paper introduces Degrees of Freedom (DoF), a continuous framework that measures how much decision-making is delegated to agents in large language model–based multi‑agent systems. By varying DoF, the authors explore a spectrum between fully centralized and fully decentralized coordination, finding that intermediate settings often outperform both extremes across several task domains. The study shows that task characteristics and retained experience influence how coordination granularity affects performance, offering empirical guidance for designing hybrid coordination regimes.
By Yiqing Peng, Hehai Lin, Zixuan Wang, Dongyi Zhang, Zhengjie Zhu, Yi Han, Yangyang Zheng, Hao Wang, Qian Li, Chengwei Qin
The paper presents a framework for optimizing AI system configurations using Gaussian process bandits with selectively verified feedback. It introduces a fixed audit design that balances reliability coverage and reward separation, derives finite‑time regret bounds under a hard audit budget, and proposes a safe adaptive extension that mixes this design with Chernoff allocations. Experiments on synthetic data compare the certified designs to adaptive heuristics and standard GP controls, showing that safe adaptive allocation can reduce aggregate regret but may have loose finite‑budget error certificates.
By Shuang Wu
GIFT‑Eval‑CTX is a new benchmark for contextual forecasting that tests whether models can use verified future information to produce accurate time‑series predictions. It builds on GIFT‑Eval and contains 10,000 instances across 20 datasets, each annotated with macro‑level and micro‑level effects derived from actual future events. The benchmark evaluates traditional TSFMs, advanced LLMs, and agentic systems, showing that structured effects can substantially lower MASE for models like GPT‑5.6 Sol and Claude Opus 4.8, with MoiraiAgent achieving the best overall performance.
By Xu Liu, Hanshu Yan, Taha Aksu, Quang Pham, Zhongzheng Qiao, Jun Hao Liew, Silvio Savarese, Doyen Sahoo, Junnan Li
The paper introduces FracLearn, a framework that enables language models to design physical structures by learning from simulated fracture experiments. In a Tetris-inspired fracture game, the model proposes architectures, gathers experience from simulated crack responses, abstracts observations into conditional principles, and generalizes these principles to create improved designs. The approach achieves a 11.0% performance gain over prior records and demonstrates that learning can occur through external knowledge and decision-making without updating model weights.
By Bo Ni
ProtoArena is a new black-box benchmark that tests AI agents on 80 cross‑component vulnerability‑exploitation environments across 11 protocol and application domains and 7 attack categories. It evaluates agents at three information levels—guided reproduction, exploitation with context, and effect‑directed black‑box exploration—using a hybrid verifier that combines rule‑based checks with LLM review of traffic and logs. In experiments, Claude Opus 4.8 achieved the highest success rates, reaching 48.8% under guided conditions and 12.5% in effect‑directed black‑box scenarios, while also demonstrating significant DNSBomb optimization performance.
By Fasheng Miao, Zhaoyu Hu, Xiang Li, Changqing An, Jiutian Zeng, Longtao Huang, Haixin Duan, Jilong Wang
DriveWLA is a unified World‑Language‑Action model that integrates semantic understanding, future scene dynamics, and trajectory generation for autonomous driving. It employs a Mixture‑of‑Transformers architecture with three experts—World, Language, and Action—connected by Cross‑Expert Attention to align semantic and dynamic representations. After reinforcement learning refinement, DriveWLA outperforms existing VLM‑based and world‑model‑based planners on NAVSIM benchmarks and generalizes well to other datasets without fine‑tuning.
By Shuzheng Wang, Haoxuan Li, Shengzhen Li, Chenxi Wu, Yajing Sun, Lu Hou, Wenyu Liu, Xinggang Wang
The paper introduces Orchestra, a robotic agent that separates task planning from action generation to follow multimodal, long‑horizon instructions. Orchestra decomposes instructions into ordered subtasks, tracks progress, updates object locations, and uses a progress evaluator to decide when to advance. On the new RMBench‑Compose benchmark, Orchestra outperforms prior methods, achieving higher success rates and longer average subtask sequences, and shows improved performance on real‑world multi‑step tasks.
By Youpeng Wen, Gang Wang, Tianyu Huang, Weiwei Zhang, Zhijian Huang, Wei Chen, Xieyuanli Chen, Yunhui Liu
MemoryVLN introduces a memory-driven navigation system that compresses long navigation histories and reuses past spatial experience. It uses forgetting-curve-inspired temporal sampling for long sequence compression, an ego-aware memory representation that fuses 2D semantic, 3D geometric, and 6D pose tokens, and a hierarchical memory bank for in-episode and context-level memory management. The system retrieves top‑K relevant memories to augment current visual tokens, achieving significant improvements on R2R‑CE and RxR‑CE benchmarks.
By Xueyao Wei, Yifei Wang, Delun Li, Shaoan Wang, Junzhi Yu, Wenjie Song, Xingyu Chen
JevGPT is a dual‑system architecture for autonomous research that combines fast evidence processing (System One) with deliberative research planning (System Two). System One quickly locates and evaluates relevant passages, while System Two sets evidence objectives and integrates findings into subsequent research steps. Experiments on question answering and document ranking demonstrate that JevGPT outperforms comparable agents, achieving faster completion, lower cost, and broader source coverage.
By Yasheng Sun, Yanning Dai, Yimeng Chen, Nanbo Li, Firas Laakom, Yifan Xu, Piotr Pi\k{e}kos, Mateusz Ostaszewski, Wenyi Wang, Cheng Luo, Yuhui Wang, J\"urgen Schmidhuber
SimInter is a pipeline that enriches user profiles for conversational recommender systems by conducting synthetic interviews between two large language model agents, grounded in users’ statistical profiles and historical conversational evidence. An audit agent flags potential fabrication and stereotyping at each turn before enrichment. In experiments on movie recommendation, SimInter significantly increases the diversity of expressed preference aspects and improves simulator performance with two recommenders, with high human agreement on enriched attributes and auditor decisions.
By Erfan Shahabi, Amirhossein Roshandel, Mohammad Sadegh Sadeghi, Azadeh Shakery
The paper introduces ALIGN, a lightweight framework that automatically generates enriched interfaces to reduce agent‑environment misalignment in large language model agents. By augmenting static environment information and step‑wise observations, ALIGN improves performance across diverse tasks such as embodied navigation, web browsing, and tool use, achieving up to a 45.67% success rate increase in ALFWorld. The method works without altering agent logic or environment code and generalizes across different agent architectures and LLM backbones.
By Kaiming Liu, Xuanyu Lei, Ziyue Wang, Peng Li, Yang Liu
The paper introduces Multi-agent Undercover Gaming (MUG), a protocol that extends the Multi-Agent Debate (MAD) framework to address hallucinations in large language models. MUG reframes MAD as a detection task for ‘undercover’ agents—those that hallucinate—by using multimodal counterfactual tests that modify reference images to reveal hallucinations. The approach enhances factual verification, cross‑evidence reasoning, and active discussion among agents, aiming to improve reliability in multimodal reasoning.
By Dayong Liang, Xiao-Yong Wei, Linhui Xiao, Changmeng Zheng
Agentic Architect is a new framework that uses large language models to automatically evolve and optimize CPU microarchitectural designs. Starting from a seed design, the LLM proposes code changes, which are compiled, simulated, and evaluated against performance, area, and power constraints, with results feeding back into subsequent iterations. The framework was tested on prefetching, cache replacement, and branch prediction, achieving notable speedups and area reductions compared to state‑of‑the‑art designs.
By Vasilis Kypriotis, Alexander Blasberg, Alexandros Moiras, Dimitrios Skarlatos
The paper introduces Verifiable Process Rewards (VPR), a framework that uses symbolic or algorithmic task-solving oracles as action-level verifiers to provide dense rewards for intermediate actions in agentic reasoning tasks. VPR is applied to Sokoban, Sudoku, and Minesweeper, and improves policy performance over outcome-reward baselines. The authors also demonstrate VPR’s effectiveness in a tool‑use benchmark, highlighting its reliance on reliable task oracles and the difficulty of extending it to less structured environments.
By Huining Yuan, Zelai Xu, Huaijie Wang, Xiangmin Yi, Jiaxuan Gao, Xiao-Ping Zhang, Yu Wang, Chao Yu, Yi Wu
The paper introduces A-LEMS, a framework that measures AI energy consumption at the goal level rather than per model invocation, defining Energy per Successful Goal (EpG) as total workflow energy divided by successfully completed goals. It also proposes the Orchestration Overhead Index (OOI) to quantify the extra energy cost of agentic systems compared to linear execution, revealing significant overheads across various hardware platforms and task families. The study demonstrates that traditional per-inference metrics underestimate the true energy cost of completing user goals in agentic AI workflows.
By Deepak Panigrahy, Aakash Tyagi, Stephen Abkin, David Kebo Houngninou, Rabi Mahapatra
The paper introduces representational commitment, a metric that quantifies hidden‑state similarity across repeated runs of a tool‑using LLM agent at the same step. Experiments on Llama‑3.1‑70B with ReAct on HotpotQA and other models show that higher step‑4 similarity correlates with lower overall trajectory variability, indicating when agent behavior stabilizes. The study also demonstrates that this metric captures stability rather than correctness, and explores monitoring and intervention techniques that can predict and reduce trajectory variability, though they do not significantly improve accuracy.
By Aman Mehta, Anupam Datta
Litmus is a zero-label system that automatically designs evaluation and monitoring metrics for AI pipelines by extracting evaluation intent from source code and targeted interrogation. It identifies what must be measured and why, then generates a justified, per-stage metric portfolio. In experiments on three real code-defined pipelines, Litmus outperforms AutoMetrics and DynamicRubric baselines in concern coverage, stage span, redundancy, and validity against per-row quality labels, especially on scientific QA.
By Prajjwal Gupta, Prasang Gupta, Vishal Bhutani, Apoorva Sharma, Sumanth Chundru, Waqar Sarguroh, Kevin Paul
MIND is a lightweight defense framework that protects memory‑augmented LLM agents from memory injection attacks by using an intent‑aware Information Bottleneck to extract compact intent–behavior representations. The approach filters out task‑irrelevant and repetitive information while preserving attack signals, enabling a lightweight detector to identify malicious memories without repeated LLM auditing. Experiments show that MIND significantly lowers attack success rates—reducing mean ASR‑r and ASR‑a by 58.1% and 53.5% on ReAct‑StrategyQA—while maintaining task accuracy and inference efficiency.
By Dongyi Liu, Haixing He, Xiaobao Wu, Jia Li
CockpitHAT is a hierarchical attribution framework for embodied multi‑agent systems that replaces positional windows with dependency‑distance thresholds derived from interaction DAGs, incorporates multi‑channel evidence through an embodied adapter, and applies a safety‑uplift to high‑risk failures during confidence‑weighted analyst consensus. The authors also release CockpitBench, a benchmark of 212 annotated failure traces labeled with ISO 26262 ASIL severity. On both the public Who&When benchmark and CockpitBench, CockpitHAT outperforms the text‑only state‑of‑the‑art ECHO, achieving higher agent‑level and step‑exact accuracies across multiple splits.
By Wei Wang, Shuanghe Liu, Zhu Zhuo, Jiaqi Zhong, Xiaozhao Zhao, Xiaojie Zuo, Jie Su