The paper introduces Harness-Aware Distillation (HAD), a method for training smaller language model agents that preserves the surrounding harness—software managing context, tools, and feedback—while focusing distillation on the teacher’s contributions beyond the harness. HAD combines an action preference that contrasts teacher actions with and without harness information, and a validity check that filters out contradictory preference pairs. Experiments on long-horizon agent benchmarks show that HAD outperforms standard on‑policy distillation, reducing unproductive loops and improving error recovery without requiring task rewards or future information.
By Moonseok Choi, Taehong Moon, Giung Nam, Juho Lee
OpenBox is a two‑stage automatic annotation pipeline that uses a 2D vision foundation model to associate 2D image cues with 3D point clouds. It then classifies instances by rigidity and motion state to generate adaptive bounding boxes using class‑specific size statistics, eliminating the need for self‑training. Experiments on Waymo Open Dataset, Lyft Level 5 Perception, and nuScenes show improved accuracy and efficiency over existing baselines.
By In-Jae Lee, Mungyeom Kim, Kwonyoung Ryu, Pierre Musacchio, Jaesik Park
The paper introduces CoC‑Seduce, a multi‑agent adversarial benchmark for evaluating how well large language models (LLMs) adhere to the rules of the tabletop role‑playing game Call of Cthulhu when acting as adjudicators. Using three LLMs to generate 5,376 samples across diverse settings and skill categories, the study benchmarks 22 target adjudicators and finds that newer releases and explicit reasoning do not guarantee robustness, with pseudo‑logic framing emerging as the most effective manipulation. The benchmark highlights the vulnerability of LLM adjudicators to rhetorical injection attacks that exploit narrative framing to bypass rule enforcement.
By Weiying Chen, Junlong Shen, Zhanyuan Guo, Xiaoou Zhou
The paper investigates whether large language model (LLM) agents that yield to a unanimous majority in multi‑agent debate truly change their underlying premise or merely adjust their statement. Using two‑hop factual questions with an unstated intermediate entity, the authors show that agents that concede still encode the original bridge in their internal representations, as revealed by a Jacobian‑lens analysis, even when the majority answer is wrong. Experiments across several open‑weight models demonstrate that hiding or removing the agent’s earlier answer increases conformity and that the original premise can be recovered from the question alone.
By Ziang Ni, Peng Zou
The paper introduces PlanPool, a method for agentic Text-to-SQL systems to manage clarification questions by maintaining a mutable question pool. By requiring agents to explicitly ask or drop each planned question, PlanPool improves ambiguity coverage and reduces silent failures compared to unconstrained or prompt-based approaches. Experiments on benchmarks derived from BIRD-Interact and Spider show that PlanPool achieves competitive execution accuracy while better handling underspecification.
By Wen-Zhi Li, Yue Gong, Konstantinos Kanellis, Balakrishnan Murali Narayanaswamy
PowerBench is a new evaluation framework that measures how language models respond to power‑shifting requests, distinguishing self‑empowerment, disempowerment, and power grabbing while controlling for neutral requests. The authors built and released a dataset covering different power domains, contexts, scales, and prior power standings, and tested 24 models from US and Chinese developers across three experimental conditions. Results show models are more likely to refuse power‑grabbing than disempowerment, and that biases exist toward helping US users gain power while resisting US users taking power from others, with additional effects when the user is an AI agent or when requests are in different languages.
By Nicolas Martorell, Wendy Brau, Gonzalo A. Heredia, Tom\'as Pablo Korenblit, Gaspar Labasti\'e, Tom\'as Gimenez Molina
Tropical Reinforcement Learning replaces the traditional sum of probabilities with a maximum operation, forming a tropical semiring. This change allows the value of a state to reflect the log-probability of its most likely verified solution and provides an explicit path that can be replayed. The proposed TROPIC algorithm, applied to deterministic, resettable environments, outperforms strong on‑policy baselines on tasks such as Sokoban, Countdown, FrozenLake, and WebShop.
By Arip Asadulaev, Aladin Djuhera, Karim Salta, Holger Boche, Fakhri Karray, Martin Takac
The paper evaluates two strategies for training language‑model‑based world models—fine‑tuning and Retrieval Augmented Generation (RAG)—across five diverse environments. Fine‑tuning generally yields higher rewards, while RAG is more data‑efficient. The authors propose a counterfactual error estimation for RAG, improve retrieval with a hierarchical query reformulation, and combine both approaches into a hybrid system that outperforms each method alone.
By Dhananjay Ashok, Shantanu Agarwal, Vivek Datla, Jonathan May, Alfy Samuel
The paper introduces HypoAgent, an agentic framework for refining abductive hypotheses over knowledge graphs. It treats user-provided conditions as steering operators, using a small generator for initial proposals, root-cause analysis to identify problematic branches, and a refiner agent that revises hypotheses based on diagnosis and prior proposals. Experiments on BioKG, PharmKG8k, and DBpedia50 show HypoAgent surpasses one-shot generation in both single-turn and multi-turn scenarios, including unconditional settings.
By Yisen Gao, Yixi Cai, Tianshi Zheng, Jiaxin Bai, Yangqiu Song
Prospective Hindsight (PH) is a self‑calibrating training principle for reinforcement learning that augments any retrospective base method with a signal derived from the gap between an agent’s prospective prediction and its retrospective evaluation. By weighting gradients with a stop‑gradient surprise based on this prediction‑reality gap, PH highlights samples where the agent’s self‑model is most inaccurate and encourages the policy and predictor to co‑evolve, progressively focusing on remaining blind spots. Experiments on single‑turn verifiable tasks and a multi‑turn personal‑agent task show that PH improves both task performance and calibration across model scales, addressing overconfident failures in single‑turn settings and underconfident successes in multi‑turn settings.
By Jiaxin Zhang, Xiangyu Peng, Qinglin Chen, Yu Li, Hiroaki Hayashi, Chien-Sheng Wu
WebUIProof is a new execution‑oriented benchmark for WebUI code generation that supplies structured specifications and dense, executable interaction tests across general WebUIs and 3D interactive simulations. It employs a UI‑agent harness that runs tests in a headless browser using a plan–act–observe loop to locate DOM elements, perform actions, observe changes, and verify assertions. Evaluations on eight commercial LLMs reveal frequent interaction‑based failures, especially on 3D interfaces, and demonstrate that training compact models with RL rewards from these tests improves functional completion and reduces build failures.
By Yun-Yun Tsai, Yuning Mao, Shiqi Wang, Junfeng Yang, Sinong Wang
The paper introduces GUI agents for continual game generation, presenting PlaytestArena—a benchmark of 200 browser-based game-generation tasks with rubrics for in‑play behavior—and Play2Code, an iterative framework where a game agent and a rubric‑blind GUI playtester refine games through shared memory. Play2Code achieves a 66.8% rubric pass rate, surpassing baseline methods by 37.1 and 14.6 points, and shows consistent score improvement across refinement rounds. The study demonstrates that GUI playtesting provides actionable, traceable feedback that can guide interactive code generation.
By Yixu Huang, Bo Li, Na Li, Zhe Wang, Kaijie Chen, Haonan Ge, Qingyi Si, Yuanzhe Shen, Ruihan Yang, Guangjing Wang, Hongcheng Guo
FastKernels is a benchmark comprising 384 GPU kernel tasks from 47 architectures across 8 categories, designed to evaluate kernel generation agents in realistic production settings. It covers 94.6% of HuggingFace Transformers architectures, scoring kernels on correctness, coverage, and speedup within their native execution paths. The benchmark reveals that kernel‑level speedups often collapse to modest end‑to‑end gains and that isolated performance rankings can misrepresent real‑world effectiveness.
By Gabriele Oliaro, Jaeseong Lee, Yichao Fu, Zhaoyuan Su, Owen Lu, Sreeram Vennam, May Jiang, Junli Wang, Hao Zhang, Zhihao Jia, Samyam Rajbhandari
The paper introduces BenchGuard, a framework that uses advanced language models to audit execution-based agent benchmarks. BenchGuard systematically cross-verifies benchmark artifacts through structured LLM protocols, optionally incorporating agent solutions or execution traces for deeper diagnostics. In tests on ScienceAgentBench and BIXBench Verified-50, BenchGuard uncovered numerous author-confirmed and expert-identified issues, including fatal errors that made tasks unsolvable, and demonstrated cost-effective auditing for complex bioinformatics tasks.
By Xinming Tu (Minta), Tianze Wang (Minta), Yingzhou (Minta), Lu, Kexin Huang, Yuanhao Qu, Sara Mostafavi
Co-Design Gym is a new benchmark suite that enables joint optimization of an agent’s embodiment and its control policy across diverse domains such as robotics, games, and networks. Unlike existing benchmarks that keep the embodiment fixed, this suite includes 20 environment families and over 85 co-design presets, covering tasks from manipulation and locomotion to electricity grids and F1 racing. The authors also provide a systematic evaluation of representative co-design algorithms, outlining the current state of the art.
By Aviraj Newatia, Yordan Tsvetkov, Leonard Pleiss, Andrew Spielberg, Rika Antonova
VisAudit is a new benchmark that tests multimodal agents on visual diagnosis, repair, and verification tasks. It presents agents with rendered charts and auxiliary evidence—such as source data, intended summaries, and code—to iteratively detect defects, modify the visualization, and confirm successful repairs. The benchmark includes 1,900 flawed charts across 21 types and 10 flaw categories, plus 300 correct charts, and shows that current models recover only about 47.4% of flawed charts autonomously.
By Shicheng Liu, Adam Kahirov, Qi Zhang, Zhimin Hu, Song Wang, Junhong Lin, Julian Shun, Yada Zhu
The paper investigates why training terminal agents often stalls when using a meta‑agent like Claude Opus to generate tasks and verifiers. It identifies three failure modes—benchmark invalidity, harness brittleness, and reward misalignment—and shows that prompt redesign and context extension can improve solvability by 5.6×, yet a 9B model still tops out at 81.3% mean pass@2. Adding hard tasks drops performance to 20.6%, highlighting that solvability depends on the model used.
By Xi Qin, Isabel Kurth, Xin Cui, Elin Park, Alexander Schaefer, Yaad Oren
HyperBrowseComp is a multilingual and multimodal web‑browsing benchmark featuring 423 manually authored, human‑validated questions in 13 languages. The questions are intentionally difficult, requiring users to locate obscure evidence, follow multi‑step clue chains, or inspect heterogeneous sources such as videos, scanned documents, images, or maps. The benchmark filters out easier questions by testing models without internet access and evaluates performance using provider‑native search and a shared external retrieval harness, with a human evaluation on a sample to contextualize model effort.
By Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri, Seung Hun Eddie Han, Ryandito Diandaru, Qinrong Cui, Jan Christian Blaise Cruz, Badrinath Chandana, Peerawat Chomphooyod, Ahmed Attia, Jonibek Mansurov, Emilio Villa-Cueva, Canh Duong Nguyen, Imran Turganov, Minghao Wu, Peerat Limkonchotiwat, Irina Nikishina
The paper introduces SCOUT, an offline multi-agent reinforcement learning framework that combines a generative behavioral prior with a decomposed value function for test-time action refinement. SCOUT uses optimal unified transport and Stein variational gradient descent to steer behavioral samples toward high-value regions, with the number of transport steps providing adaptive scaling. The authors prove a single-term KL bound on the joint soft-value gap under the individual-global-max principle and demonstrate that SCOUT outperforms existing methods on both discrete and continuous offline MARL benchmarks, including offline-to-online settings.
By Dongsu Lee, Haoran Xu, Amy Zhang
The paper introduces Follow the Winners (FTW), a critic‑free policy‑learning algorithm that adapts the cross‑entropy method for reinforcement fine‑tuning of agentic large language models. FTW replaces group rollouts with an ordinal filter on replay‑buffer samples, achieving polynomial concentration in the order statistic of returns and offering a bounded risk‑seeking offset that balances variance reduction. Experiments on Sokoban and Search‑R1 show that FTW matches the performance of GRPO and PPO while reducing reliance on value models or repeated rollouts.
By Joery Ari\"en de Vries, Neil David Lawrence, Zhenwen Dai