The study evaluates 66 combinations of four configurable harnesses and five language models across three benchmark suites, revealing that model rankings and best harnesses vary significantly with the task. Notably, the openJiuwen harness consistently yields the highest scores for the Kimi model, while GPT performs best with the lean PI scaffold on Terminal‑Bench 4. The findings demonstrate that a model’s performance is highly dependent on the harness it is paired with, and that higher cost or a vendor’s own harness does not guarantee superior results.
By Yixuan Li, Yiyun Zhou, Yao Long Teng, Fuchao Yang, Yanchen Deng, Zhiyi Lyu, Xuyu Dong, Feng Chen, Bo An
The paper evaluates how Large Language Models generate preference distributions for air travel, restaurants, and consumer products. It finds that while each model produces self-coherent outcomes that stabilize quickly, there is significant disagreement across different model families and scales, with little consensus even on the most probable preferences. These discrepancies persist across various decoding strategies, temperature settings, and prompt variations, indicating that the model choice itself has a larger impact than prompt wording.
By Fan Huang, Minsuk Kim, C. Tyler Diggans, Filippo Radicchi
The paper introduces InFlowOp, a label‑free optimization framework that assigns costs to each decision in a multi‑agent workflow, balancing agent competence against execution time. It determines task granularity and agent assignment before execution and corrects faults during execution using the same cost metric. The authors also present Braid, a benchmark for multi‑agent coordination, and show that InFlowOp outperforms single‑agent baselines by up to 11.97% across various domains.
By Xuehang Guo, Haoyu Wang, Shengyu Chen, Zach Chen, Wei Cheng, Qingyun Wang, Haifeng Chen
The paper introduces Independent–Communicate–Revise (ICR), a framework that isolates communication effects in large language model multi‑agent systems by fixing initial reasoning and measuring how messages influence answer revision. ICR evaluates correction, preservation, and selectivity across four reasoning benchmarks, revealing that similar overall accuracy can mask divergent revision behaviors. The study shows that richer messages can both improve and harm outcomes, and that receiver policies can shift preservation and correction dynamics differently across tasks.
By Shixuan Li, Wei Yang, Peiyu Zhang, Anzhe Cheng, Heng Ping, Paul Bogdan
The paper introduces APIVIS, a training-time framework that integrates finite-budget Gumbel search into reinforcement learning with verifiable rewards (RLVR) for mathematical reasoning. APIVIS combines direct and searched responses within each rollout group, uses selective supervision on search-improved tokens, and applies value-guided selection to improve verifier rewards at each searched state. Experiments on standard mathematical reasoning benchmarks and various model scales show significant performance gains over existing search-based methods.
By Shaohuai Liu, Yuning Wu, Haoran Liu, Enzo Jia, Devin Chen, Kai Wei
The paper introduces a typed snapshot‑settlement contract for auditing concurrent actions in large language model agent environments. It evaluates three properties—order sensitivity, useful progress, and replay consistency—across five settlement policies, using 28,800 exhaustive permutation trials and 2,160 scripted multistep episodes. Results show that joint policies are spatially order‑invariant with fixed priorities, but conservative rejection only completes 31.25% of agents in a six‑agent doorway task compared to 90.28% for random tickets, while a full‑state journal audit successfully replays 156 checkpoints and rejects 1,332 constructed corruptions.
By Haotian Chen, Bowen Ye, Yuning Zhang, Jingkun Yu
The paper explores how large language models (LLMs) can be grounded in dynamic stochastic general equilibrium (DSGE) simulators to generate and evaluate economic policy actions. By embedding an instruction‑tuned LLM within six Snowdrop‑backed DSGE models, the authors create a long‑horizon credit‑assignment problem where the model observes economic states, selects bounded policy actions, and receives simulated outcomes and rewards. They compare reinforcement learning methods—PPO, which uses a learned value function to propagate delayed rewards, and GRPO, a critic‑free baseline—to assess how well the LLM’s policy decisions align with simulated economic dynamics, testing factors such as semantic signals, reward horizons, and cross‑simulator transfer.
"whyItMatters":"The study demonstrates a method for evaluating LLM‑generated policy proposals based on their actual simulated economic consequences rather than merely their linguistic plausibility."
By Aditya Dubey, Namah Gupta, Vinti Agarwal
The paper introduces Federated Agent Optimization (FAO), a framework for enabling large language model agents to improve collaboratively while keeping raw data, trajectories, and private knowledge local. FAO treats agent capabilities—such as memory, tools, rewards, skills, and structured knowledge—as a multi‑objective optimization space that balances utility, privacy leakage, and communication cost. It outlines methods for abstracting, protecting, aggregating, and adapting private experience into transferable capabilities, and highlights key challenges and future research directions for trustworthy federated agent systems.
By Qiang Yang, Zhiqiang Kou, Xueyi Zhang, Dong-Dong Wu, Hanlin Gu, Jing Guo, Yang Liu, Di Jiang, Qian Xu
The paper introduces an educator‑guided large language model (LLM) pedagogical agent that provides scaffolded feedback for conceptual database design. Embedded in an entity–relationship diagram editor, the system bases its feedback on the student’s artifact, assignment requirements, educator‑authored rubrics, and instructional resources, using a four‑stage workflow that progresses from concept checks to low‑detail feedback and localized clarification. In a deployment across three ERD environments with 383 feedback episodes, 71.1% of target‑level changes incorporated the hidden diagnostic target, and qualitative analysis highlighted the benefits and challenges of staged disclosure.
By Sara Riazi, Pedram Rooshenas
DeFA is a dependency-guided framework that attributes failures in large language model agents by constructing an event dependency graph and a failure propagation graph from protocol relations and semantic dependencies. It identifies violating events, traces their sources and effects, and determines the decisive error, responsible agent, and error category. The method supports long trajectories through segmentation and has shown superior accuracy on text, image, and video tasks, while its diagnostic feedback can improve agent performance on subsequent tasks.
By Bo Deng, Xinlei Zheng, Yi Wei, Kang Zhou, Chongyang Tao, Renzhao Liang, Xuanren Chen, Lifan Guo, Chi Zhang
The paper introduces TRACE, a token‑level objective designed to reduce multi‑turn safety risks in large language models. TRACE assigns each token a weight based on the discounted return of a refusal‑attributable advantage, comparing a frozen reference model with a refusal‑ablated copy to credit early tokens for later refusal evidence. Evaluated across five open‑weight models and seven multi‑turn attacks, TRACE achieves the lowest attack success rate in all 35 model‑attack pairs while maintaining model utility within 1.23 points on MMLU and HellaSwag.
By Fengpeng Li, Kemou Li, Qizhou Wang, Haiwei Wu, Jiantao Zhou, Di Wang
The paper introduces Spiking Contrastive Attention (SCA), a module designed to reduce spectral bias in Spiking Transformers by enhancing high‑frequency information. It demonstrates that spiking neurons and spiking self‑attention act as low‑pass filters, leading to loss of high‑frequency components. Experiments show that SCA improves performance across image classification, semantic segmentation, and event‑based tracking while maintaining lower complexity than the original spiking self‑attention.
By Xiaoli Liu, Malu Zhang, Yang Yang
DRelay introduces a global draft context mechanism to improve prefix-aware parallel speculative decoding for large language models. By using a global reader to extract predictive information across the entire draft block and a causal selector to repair early token selection errors, DRelay extends the accepted prefix length and enhances decoding performance. Experiments on eight benchmarks show consistent gains over existing methods such as DFlash, Domino, and DSpark, with notable speedup improvements in SGLang serving.
By Zhuoyu Wang, Junnan Huang, Xinyu Chen
The paper introduces a multi‑agent large language model framework that interprets personalized health checkup results by assigning distinct tasks to specialized agents and combining their outputs. Experiments on 120 Korean compound queries show that the multi‑agent approach improves overall quality scores and user preference compared to a single‑agent baseline, though it incurs higher latency and cost. The benefits are most pronounced for queries requiring personal‑record lookup.
By HyungJun Kim, Taehan Lee, Soojin Cheon
The paper audits whether routing entropy in Attention‑Residual transformer variants (Swin‑Tiny and DeiT‑Small) trained on CIFAR‑10/100 can signal prediction uncertainty beyond model confidence. Three tests examine the presence, consistency, and predictive power of routing signals, while a sensitivity audit measures how much injected effect the probes recover. Results show no significant improvement over confidence alone, with only modest recovery of injected signals and no consistent gains across seeds or metrics.
By Wenhao Liang, Lin Yue, Wei Emma Zhang, Mingyu Guo, Olaf Maennel, Weitong Chen
OpenMTB‑Audit is an open‑source benchmark that tests large language models on 500 synthetic non‑small cell lung cancer cases, covering five adversarial error categories and four safety labels: Supported, Partially Supported, Unsupported, and Insufficient Information. The study found that all eight tested LLMs over‑refused Partially Supported recommendations, collapsing labels to achieve high safety scores. A deterministic seven‑module framework, MTB‑AuditAgent, was introduced to reduce over‑refusal to 6.7% and reach 91.2% accuracy, while an oncologist annotation study highlighted disagreement around the boundary between information sufficiency and treatment optimization.
By Negin Ashrafi, Jia Luo, Stacey M. Frumm, Roxana Daneshjou
The paper evaluates five vision‑language models on autonomous driving tasks under various visual input conditions, finding that visual corruption affects accuracy and confidence differently across models and datasets. It then tests Visual Evidence Augmentation (VEA) as an inference‑time technique to enhance reliability, observing mixed improvements depending on the model and setting.
By Manasa Mariam Mammen, Priyanka Mary Mammen, Zafer Kayatas, Stefan Wagner
The paper proposes a new framework for collaboration between a small language model (SLM) and a large language model (LLM) that treats the interaction as an information acquisition problem under an API budget constraint. Instead of delegating reasoning tasks, the SLM remains the primary reasoner and selectively queries the LLM advisor with targeted questions, using a three-stage RLVR approach to decide when to call the advisor, how to phrase queries, and how to integrate the responses. Experiments on mathematical reasoning and coding tasks show that this strategy improves the performance–cost tradeoff compared to existing baselines and can transfer to other advisor model families without additional training.
By Yongjun Kim, Xiaoxiao Li, Jaeho Lee
LineupRL introduces a reinforcement learning framework with verifiable rewards for time series captioning, using a frozen large language model to identify the correct time series from a set of distractors based on a generated caption. This approach bypasses the limitations of supervised fine‑tuning and traditional RL rewards that poorly transfer to open‑ended time series generation. Experiments on two captioning benchmarks, as well as forecasting and reconstruction tasks, show that LineupRL outperforms both SFT and RL baselines across all metrics, and its trained 3B vision‑language model surpasses a 72B model distilled from SFT captions. The method also demonstrates resistance to reward hacking and produces captions that accurately trace trends and name key values.
By Haochen Zhang, Laura Yao, Zachary Plotkin, Gengwei Zhang, Tianlong Chen
The paper introduces a continuous evaluation framework that assesses both outcome-level and process-level aspects of evolving enterprise AI agent skills. It applies this framework to two variants of a Business Value Determination skill, running 240 trials across multiple models, harnesses, and specifications. The results show that while most trials pass final numerical checks, a large majority still exhibit process-level deviations, and dependency attribution reduces the number of failed checks per run. The framework also provides reusable regression tests and highlights specification sensitivity across configurations.
By Ngoc Phuoc An Vo, Aarya Doshi, Vadim Sheinin