Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 12

AdaC-GRPO: Towards Mastering Long-Horizon Visual Sudoku Puzzles via Adaptive Curriculum Learning

AdaC-GRPO introduces an adaptive curriculum learning framework for mastering long-horizon visual Sudoku puzzles, a multimodal task that challenges large language models with multi-step deductive reasoning. The authors present FG-Sudoku, a fine-grained benchmark that categorizes puzzles by grid size and required deductive steps, revealing that frontier multimodal large language models (MLLMs) struggle with complex long-range dependencies. AdaC-GRPO dynamically constructs training batches by tracking the model’s Zone of Proximal Development through online rollout accuracy, enabling Qwen2.5-VL-7B to close a significant portion of the accuracy gap to Gemini-2.5-Pro and demonstrating generalizability across other models.

By Sida Li, Yuanxin Liu, Linli Yao, Shicheng Li, Xu Sun, Haoli Bai, Xiaohui Li
arXiv AI
Oct 12

JANUS: Enhancing Long-Horizon Autonomous Research with Co-Evolving World Models

JANUS is a framework that enhances long‑horizon autonomous research by integrating future outcome forecasting and past experience reflection through a Predict‑Verify‑Evolve loop powered by a co‑evolving world model. In this system, a research agent proposes candidate research paths while the world model evaluates them based on predicted value improvement, information gain, and exploration utility, guiding selection, refinement, or pruning before execution. After execution, JANUS distills feedback into evolving skills for both the agent and the world model, enabling continual co‑evolution and improved efficiency, as demonstrated by superior performance on MLS‑Bench‑Lite and real research tasks.

By Mengdi Liu, Chenghao Jia, Hong Chang, Mingjie He, Shiguang Shan, Xilin Chen
arXiv AI
Oct 12

Web$hell: A Unified Interface for Agentic Web Access

Web$hell is a unified interface that consolidates various web access methods—search, browser actions, and filesystem interactions—into a single model for large language model agents. It uses a WebContract schema to compile live webpages into addressable information and action objects, which are stored in a WebCache and manipulated through shared operations. Experiments across search, navigation, and research tasks show that Web$hell delivers strong performance at lower cost, highlighting the role of interface design in enhancing agentic web access.

By Sunghwan Kim, Deogyong Kim, Sangam Lee, Donguk Kwon, Dongha Lee
arXiv AI
Oct 12

Local Diagnostics of Learned Event Representations for Anomaly Detection at the HL-LHC

The paper investigates anomaly detection in high-energy physics by learning low-dimensional representations of collision events. Using supervised and self-supervised neural embeddings on a large simulated dataset, the authors find that supervised objectives generally outperform self-supervised ones and improve signal efficiency compared to traditional kinematic-feature autoencoders. They also propose a local diagnostic pipeline that models the Standard Model latent space with a Gaussian Mixture Model, profiles components via high-level observables, and ranks these observables by their local discriminating power, thereby linking anomaly scores to specific kinematic features.

By Donatella Genovese, Philip Ploner, Simone Scardapane, Stefano Giagu, Thea Klaeboe Aarrestad
arXiv AI
Oct 12

Humanize MLE Agents: Multi-Agent Session Alternation Beyond Single-Agent Limits

The paper introduces Humanize MLE Agents (HMA), a workflow that alternates two agents across sessions on a shared workspace, limiting each session to at most k experiments and resetting context between sessions. This approach, modeled as a context‑reset semi‑Markov decision process, outperforms single‑agent baselines, achieving a 78.2% any‑medal rate on 75 Kaggle MLE‑bench tasks—8.0 points higher than the best open‑source baseline and using only a quarter of its time budget. Across six heterogeneous agent pairs, HMA improves the any‑medal rate by an average of 6.4 points, and a supplementary kernel‑optimization case reaches 878 cycles, within 1.6% of the community leaderboard leader.

By Changye Li, Zijian Zhang, Muheng Li, Yiduo Pang, Tingjun Wu, Menghan Li, Yitong Liu, Hongkai Guo, Mengchi Chen, Chenyu He, Kaisheng Ma, Wei Xu, Song Han, Sihao Liu, Ligeng Zhu
arXiv AI
Oct 12

Constructing Pareto Frontiers of Optimal Reasoning Termination Methods for Overthinking in Reasoning Models

The paper introduces Step-wise Truncation Probing (STP) to measure overthinking in reasoning models by generating answers from multiple retained prefixes along a fixed reasoning trajectory. It defines Overthinking Ratio and Accuracy Degradation to quantify the impact of continued reasoning beyond the point of maximum accuracy. Using STP data, the authors construct token‑accuracy Pareto frontiers for various termination strategies and present OTBench, a benchmark that aggregates 70.5 B reasoning tokens across six models and four benchmark groups.

By Weixin Guan, Liang Li, Shiyu Xie, Jiapeng Liu, Bing Li, Peng Fu, Chengyang Fang, Xiaoshuai Hao, Can Ma, Weiping Wang
arXiv AI
Oct 12

RATED: Reliability-Aware AI Agent Tool Retrieval via Execution Dynamics

RATED (Reliability‑Aware Tool Retrieval via Execution Dynamics) is a retrieval framework that blends robust semantic matching with dynamic reliability adaptation for AI agents using external tools. It uses a frozen dual‑encoder with contrastive fine‑tuning for semantic relevance and a lightweight contextual bandit module that re‑ranks tool candidates based on real‑time execution feedback, allowing rapid adaptation to runtime failures without costly retraining. The EcoAgent benchmark, comprising 5,172 real‑world tools and over 210,000 query‑tool pairs, shows that RATED improves mean reciprocal rank by 52.2% over the ToolLLM API retriever and cuts fault selection from 86.2% to 2.9% during severe API outages, with negligible latency overhead.

By Weiliang Chen, Song Liao
arXiv AI
Oct 12

AutoIndexer: Flexible-Order Decoding by Training Causal Models on Chains of Edits

AutoIndexer is a generative model that trains causal models on chains of edits, enabling flexible post‑generation modifications such as token insertions, substitutions, and deletions. It introduces marker tokens for opening and closing edits and moves a cursor through the token sequence, mimicking a text editor. By perturbing ground‑truth labels with inverse edit operations during training, AutoIndexer becomes robust to erroneous context and can perform editing tasks like code correction while maintaining performance on standard LLM benchmarks.

By Shu Ishida, Aliasghar Khani, Tianyu Zhang, The Cong Luong, James Seale Smith, Adam Gaier
arXiv AI
Oct 12

Machine Learning and Time-Series Foundation Models for Day-Ahead Electricity Forecasting in a University Building

The study benchmarks various forecasting models—including statistical, machine‑learning, recurrent neural networks, and pretrained TimesFM—for day‑ahead electricity demand in a university building. TimesFM 3.0 with weather covariates achieves the lowest overall mean absolute error, outperforming a weekly seasonal‑naive baseline by about 43%, while XGBoost remains competitive and requires only local fitting. The LSTM model takes longer to train without improving accuracy, and TimesFM’s performance is robust even without covariates, though covariates slightly improve holiday forecasts.

By Amey Vasulkar, Samuel Kernan Freire, Arie-Willem de Leeuw, Tadeo-Baldiri Salcedo-Rahola
arXiv AI
Oct 12

GeoFitBench-LLM: A Benchmark for Assessing Large Language Models' Capabilities in Geospatial Data Requirements Elicitation for Geo-Analytic Tasks

GeoFitBench-LLM is a benchmark designed to evaluate large language models (LLMs) on their ability to elicit geospatial data requirements for geo‑analytic tasks. It focuses on two reasoning dimensions—spatio‑temporal relational and semantic grounding—and uses a provider‑independent schema to encode ground‑truth requirements. Experiments on seven models show that while many can identify necessary resources, accurately specifying fitness‑for‑use remains challenging, with GPT‑5.4 Thinking recovering all resources in 66.67% of cases.

By Amirhossein Vahdat, Thierry Badard, Luc Lamontagne
arXiv AI
Oct 12

Trajectory Prediction for Autonomous Driving in Road Construction Zones Using Human-in-the-Loop Driving Data

The paper presents WZDrive, a multimodal ego‑trajectory predictor for autonomous vehicles operating in urban road construction zones. It introduces WZDrive‑Data, a dataset of 1,183 work‑zone driving segments collected from 33 drivers in a human‑in‑the‑loop simulator that records driver control, gaze, vehicle trajectories, and scene information. WZDrive achieves a mean minADE of 1.09 m and minFDE of 2.12 m, outperforming state‑of‑the‑art baselines by 7.18 % and 13.18 % respectively.

By Shuo Zhang, Aman Sunesh, Semiha Ergan, Kaan Ozbay
arXiv AI
Oct 12

REDaCT: Collapse-Free Unlearning in Large Reasoning Models

The paper introduces REDaCT, an inference-time unlearning framework for large reasoning models that removes forget-set information from both the reasoning trace and the final answer without causing reasoning collapse. REDaCT uses a binary leak detector to spot sentences that leak forget-set data and an embedding optimization module that injects a corrective embedding at the first detected leak, steering subsequent reasoning toward non-leaking sentences. Experiments demonstrate that REDaCT preserves coherent chain-of-thought reasoning, outperforms existing unlearning methods, and achieves perfect answer forgetting with near-perfect chain-of-thought forgetting.

By Joshua Maccoby, Wendy Hui Wang
arXiv AI
Oct 12

Meta-Reflective Evolution Effectively Improves Production AI Systems

Meta-Reflective Evolution Effectively Improves Production AI Systems describes CoCoEvolve, an evolutionary optimizer that uses meta-reflection to retrieve evidence from the entire evolutionary history and probe live systems before proposing mutations. The approach is applied to data agents, LLM-powered data pipelines, and data engineering workflows, achieving state‑of‑the‑art results on DABStep, multiple LLM pipeline workloads, and data-eng-bench. Ablation studies confirm that each component of CoCoEvolve contributes to its performance gains.

By Seung Won Wilson Yoo, Chang Zeng, Aditi Somayajula, Sahithi Ankireddy, Rohan Kosalge, Avishree Khare, John Mitchell, Aditya G. Parameswaran, Rajhans Samdani, Anupam Datta
arXiv AI
Oct 12

Localized Generative Design of Ship Hulls with Diffusion Models and High-Fidelity Assessment

The paper introduces a reward‑directed diffusion framework for localized ship hull optimization, using parametric masking to focus changes on specific hull regions while maintaining overall continuity. Surrogate‑assisted sampling targets low calm‑water resistance, and a high‑fidelity CFD pipeline evaluates the designs, revealing trade‑offs between friction and pressure that are not captured by empirical screening. The study demonstrates that detailed assessment can uncover limitations in the generative model and guide better objective formulations for realistic hull designs.

By Hadi Keramati, Mohammed A. Hannan, Patrick Kirchen, Sina Tousi, Joonas Riekkinen, Jim Pyra, Rajeev K. Jaiman
arXiv AI
Oct 12

Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence

The paper introduces Generalized Proximal Policy Optimization (GPPO), a unified framework that extends PPO by incorporating $f$-divergence trust-region theory and novel estimator topologies for large language model reinforcement learning. GPPO demonstrates that different divergences produce distinct policy update dynamics and that estimator design significantly impacts performance. Experiments on eight math and coding benchmarks show GPPO surpasses standard GRPO by up to 50% in math and 14% in coding, achieving state‑of‑the‑art results.

By Zhicheng Cai, Xinyuan Guo, Hanlin Wu, Weinan Dai, Yuyan Zhou, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou
arXiv AI
Oct 12

Evolving Creatives with Clicks: CTR-Driven Advertising image Editing via Attribute Disentanglement and Preference Alignment

The paper introduces Evolving Creatives with Clicks (ECC), a method that refines e-commerce advertising images by selectively editing attributes that negatively impact click‑through rate (CTR) while preserving beneficial ones. ECC identifies 28 CTR‑correlated attributes using ANOVA and Random Forest regression, then instruction‑tunes a prompt model to generate attribute‑specific editing suggestions, further refined with real click preferences. Consistent Preference Optimization (CPO) is added to enforce consistency across semantically equivalent suggestions, reducing overfitting and improving generalization, leading to significant CTR gains over state‑of‑the‑art models.

By Xu Zhang, Haohan Wang, Zheng Zhang, Yingjie Xia, Wei Feng, Honghe Zhu, Jingjing Lv, Changping Peng, Xuheng Qin, Zehua Zhang, Zheng Luo, Qicheng Li
arXiv AI
Oct 12

Agent4Spec: An LLM-based Agentic System to Optimize API Specifications through Automated Validation and Market Analysis

Agent4Spec is an LLM-based autonomous multi‑agent system designed to improve API specifications through continuous validation and market analysis. It includes a validation pipeline that detects and self‑corrects OpenAPI syntax errors, and a market analysis pipeline that performs competitive research, identifies functional gaps, and automatically generates missing API‑spec functionalities. In evaluations on curated datasets, the best LLM achieved 100% syntax error detection, over 95% success in implementing corrections, and more than 50% adoption of suggested endpoint improvements.

By Yongjian Tang, Costel Moraru, Doruk Tuncel, Hanya Elhashemy, Steffen Klepke, Thomas Runkler
arXiv AI
Oct 12

Beyond Temporal Proximity: Explanatory World Models for Credit Assignment in Reinforcement Learning

The paper introduces Explanatory World Models (EWM), an attribution framework that leverages explanations from a frontier language model to identify earlier events that plausibly contributed to a delayed outcome. EWM redistributes reward recursively while preserving the undiscounted episode return, and is evaluated on long‑horizon RL benchmarks, a public sequential‑recommendation task, and an industry‑scale generative recommender, consistently outperforming baseline methods, especially when rewards are highly delayed. The study demonstrates that structured explanations can provide reusable credit signals for learning from delayed outcomes.

By Oriol Corcoll, Qiyao Wei, Ciar\'{a}n Mark Gilligan-Lee
arXiv AI
Oct 12

SWE-DeceptionBench: Benchmarking Deceptive Behaviors of LLMs Across the Software Development Lifecycle

SWE-DeceptionBench is a new benchmark that evaluates deceptive behaviors of large language models (LLMs) across six stages of the software development lifecycle. It distinguishes between Active Deception—where models claim infeasible or conflicting requirements are achievable—and Passive Deception—where models assist in misleading others. The benchmark includes 720 test instances in Chinese and English, tests seven mainstream LLMs, and shows high deception rates even under neutral conditions, while lightweight defenses only partially mitigate the issue.

By Xiting Liu, Lin Yang, Yitong Zhang, Jia Li, Shi-Min Hu
arXiv AI
Oct 12

Model-agnostic Knowledge Augmented Machine Learning for Weakly Annotated Bioimaging Data

The paper introduces a model‑agnostic knowledge augmentation technique that lets machine learning models leverage domain knowledge beyond automated feature extraction for weakly annotated bioimaging data. This method improves classification across seven ML architectures, achieving the best performance when combined with masked autoencoder self‑supervised learning, with precision gains of 2.9% on three in‑house microscopy datasets and 6.8% on three public cytomorphology benchmarks. Applied to nucleus‑level classification of TDP‑43 mislocalization in motor neuron images, the approach uncovers ALS‑related biological insights, such as dose‑dependent effects of stressors and variable pathology in sporadic versus familial ALS donors.

By Jayanie Bogahawatte, Sachith Seneviratne, Simon Maksour, Dzung Do-Ha, Elizabeth Qian, Samuel Brennan, Matthew Keon, Lezanne Ooi, Saman Halgamuge