Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 9

TestJack: Should you trust the results in coding benchmarks? Agentic Coding Benchmarks Auditing via Evaluator Evolution

TestJack is a scalable framework that audits large language model (LLM) coding agents by generating new tests for each trial, focusing on prompt requirements that may be violated. It keeps only the tests passed by the ground‑truth patch and re‑examines failures, providing replayable evidence for each confirmed error. In experiments across six model backends and five benchmarks, TestJack found that 34.4% of trials deemed correct actually violated task requirements, reducing the overall resolution rate from 50.6% to 33.2%.

By Shuangjie Yao, Hao Wang, Koushik Sen, Simin Chen, Baishakhi Ray, Dawn Song
arXiv AI
Oct 9

SAVU-BENCH: A Real-World Benchmark for Spatial Audio-Visual Understanding

SAVU-Bench is a real‑world benchmark designed to evaluate spatial audio‑visual understanding across three capability levels and seven tasks, addressing limitations of prior simulated‑scene datasets. It includes a diagnostic set, SAVU‑Diag, that breaks down reasoning questions into grounding and alignment sub‑tasks, revealing that many errors stem from failures in these prerequisites. A training‑free baseline, SAVU‑EA, improves spatial grounding and joint matching but high‑level reasoning remains difficult, underscoring the need for better audio perception and cross‑modal spatial integration.

By Yu Chen, Ruihang Liu, Yangguang Xu, Xinyue Jiang, Mohammed Bennamoun, Farid Boussaid, Xinyuan Qian, Qiuhong Ke
arXiv AI
Oct 9

Grammar Concept Annotation at Scale: Deployed Fine-Tuned Small Language Models Outperform Prompted Frontier Models

The paper reports that fine‑tuned small language models (SLMs) can outperform large frontier models in annotating grammar concepts from learner–tutor transcripts. By training a 0.8B Qwen3.5 model on teacher‑generated data and embedding the annotation rules into adapter weights, the authors deploy an efficient end‑to‑end grammar mastery tracker that matches or exceeds the performance of 4B reference models and GPT‑5.4/5.6 on two human‑curated benchmarks. The lightweight model also cuts serving costs by about 16× and boosts learner engagement and key business metrics in an online experiment.

By Marjan Celikik, Ana Peleteiro Ramallo, Javier Morales
arXiv AI
Oct 9

AffordDrive3D: Affordance-Aware World-Action Modeling with Spatial Understanding

AffordDrive3D is a world-action model that jointly learns future action-relevant regions and spatial geometry for autonomous driving. It uses a VLM backbone to forecast drivable areas and collision-critical regions while predicting future geometry from RGB world-model latents. On the NAVSIM benchmark, it achieves state‑of‑the‑art performance with 91.3 PDMS and 89.9 EPDMS, showing the benefit of combining affordance prediction with geometry modeling for trajectory planning.

By Tianhui Cai, Xinglong Sun, Chao Fang, Zhenxin Li, Rui Song, Jose M. Alvarez, Yunxiang Mao, Jiaqi Ma, Langechuan Liu
arXiv AI
Oct 9

ActiveMedAgent: Cost-Aware Trajectory Learning for Multimodal Medical Diagnosis

ActiveMedAgent is a cost‑aware framework that guides multimodal medical AI to acquire diagnostic evidence sequentially, mirroring clinical practice of escalating from cheap to costly tests. It uses a frozen vision‑language model to track diagnosis probabilities, scores each potential acquisition by diagnostic utility minus cost, and trains a lightweight MLP controller offline to decide when to request more evidence or to stop. Experiments on three benchmarks show that trajectory‑based policy learning outperforms unguided acquisition and full‑modality baselines, revealing an information overload effect where the agent can correctly diagnose with fewer channels than the full‑modality approach.

By Weiwei Ma, Xiaobing Yu, Peijie Qiu, Jin Yang, Zhaoqi An, Xuanzhao Dong, Xiaoqi Zhao, Xiaofeng Liu
arXiv AI
Oct 9

EvoKnow: Continual Knowledge Evolution for AI-Generated Image Detection

EvoKnow is a replay‑free framework for continual AI‑generated image detection that treats forensic knowledge as evolving. It keeps a shared forensic basis from base domains, adds isolated residual experts for new generators, and uses an Analytical Incremental Router (AIR) updated in closed form from current‑stage data and accumulated statistics. Experiments show strong cross‑generator generalization, few‑shot expansion, and low forgetting, achieving 96.70% accuracy on non‑base generators and 96.32% mean stage‑wise accuracy under a strict replay‑free protocol.

By Zhiheng Peng, Wenwei Jin, Yangshi Ge, Siyu Xia, Jiawei Li, Xu Tang
arXiv AI
Oct 9

Rewiring Semantics, Dynamics, and Control: A Simple yet Effective Action-Centric Tri-Stream Transformer

The paper introduces ACT^3, an Action-Centric Tri-Stream Transformer that merges semantic insights from Vision‑Language Models with dynamics predictions from World Models to generate control actions for robotic manipulation. By allowing a dedicated action expert to attend to both VLM and WM representations while keeping each backbone’s forward pass independent, ACT^3 preserves the distinct roles of context streams. Experiments on simulated and real‑world benchmarks demonstrate that ACT^3 outperforms existing approaches.

By Shuang Luo, Yilun Kong, Yunpeng Qing, Yihang Jiao, Zhi Hou, Shunyu Liu, Xiaogang Wang, Dacheng Tao
arXiv AI
Oct 9

RoboAware: Learning to Coordinate Embodied Skills from Counterfactual Outcomes

RoboAware learns to coordinate modular robot skills by training a state‑conditioned responsibility coordinator using counterfactual outcomes. It introduces State‑Locked Counterfactual Branching (SCB) to generate outcomes for all admissible skill families from the same state, and Execution‑Aware Learning (EAL) to distill these outcomes into family‑conditioned values via Monte Carlo tree search and Q‑learning. In single‑episode evaluations across 100 tasks, RoboAware achieves a 77.0% overall success rate, outperforming existing baselines on RoboSuite, LIBERO‑Pro, and RoboTwin bimanual tasks.

By Bohan Zhou, Xingbei Chen, Emily Huang, Weilin Ruan, Haojian Huang, Yehang Zhang, Zexi Li, Wenqian Li, Qize Yu, Zetian Song, Leyi Wu, Jinghao Li, Mingxuan Song, Xinrun Xu, Zongyang Qiu, Yangkai Wei, Tianyi Zhang, Kaiwen Zhou, Yinchuan Li, James Cheng
arXiv AI
Oct 9

Design Creativity Bench: Measuring creativity in LLM-Generated UI

The paper introduces Design Creativity Bench, a benchmark for evaluating creativity in large language model (LLM)-generated user interface (UI) designs. It measures originality (distinctiveness among models on the same prompt), creative range (how designs vary across prompts for the same UI goal), and appropriateness (how many brief criteria each design meets). Results show LLMs produce designs that are generally appropriate but far less original and varied than human designs, indicating a need for stronger measures to address this limitation.

By Aman Rusia, Abhijit Bhole, Prashank Gupta, Dipanjan Dey
arXiv AI
Oct 9

Timer-M1: A Multivariate Time Series Foundation Model via Learning Primitives

Timer‑M1 is a pretrained multivariate time‑series foundation model that learns with primitives for zero‑shot forecasting. It uses a primitive‑based data synthesis and pretraining pipeline to generate series with shared temporal and relational patterns, assembling them into multivariate samples with distinct channel roles. The model employs gated two‑dimensional Transformer blocks that dynamically allocate cross‑variate attention, achieving top performance on several large‑scale forecasting benchmarks.

By Haoran Zhang, Haixuan Liu, Xingjian Su, Yong Liu, Zhi Chen, Yuxuan Wang, Jianmin Wang, Mingsheng Long
arXiv AI
Oct 9

A Geometric Approach to Soft Actor-Critic with Zonotopes for Locomotion Learning

The paper introduces GeZo‑SAC, a variant of Soft Actor‑Critic that incorporates geometric representations (zonotopes) to adapt critic pessimism based on state and action. Each critic predicts a zonotope whose width is subtracted from the critic value as a pessimistic offset, and the disagreement between critics is measured and used to blend the critics via a log‑sum‑exp aggregation. Experiments on MuJoCo‑v5 locomotion tasks show GeZo‑SAC achieving the highest mean return on Ant‑v5 and Hopper‑v5, while also reducing actuator work and action effort and maintaining near‑zero overestimation frequency.

By Panagiotis Roditis, Panagiotis P. Filntisis, Petros Maragos
arXiv AI
Oct 9

ReSI: Recursive Safety Improvement toward Resistant and Resilient AI

ReSI is a recursive safety improvement framework that iteratively applies diverse red‑teaming methods to identify vulnerabilities in AI models, develops training recipes, and selects the update with the greatest safety gain while retaining capability. Across four dense and mixture‑of‑experts models, ReSI matches or surpasses frontier models on both in‑distribution and out‑of‑distribution safety benchmarks, and outperforms alignment baselines on nearly all safety evaluations. The framework notably reduces the mean X‑Teaming attack success rate from 86.01% to 31.45%, outperforming GPT‑5.6‑Luna’s 56.69% and demonstrating stronger resilience to unseen attacks.

By Jingnan Zheng, Dongcheng Zhang, Yi Zhang, Ming Zhang, Qiaosheng Zhang, Youbang Sun, An Zhang, Xiangnan He, Tat-Seng Chua, Xia Hu, Bowen Zhou, Chaochao Lu, Xiang Wang
arXiv AI
Oct 9

VibeEdit: Image Editing with Canvas Instructions

VibeEdit is a new image‑editing tool that lets users draw spatial marks and add brief notes directly on an image to create a canvas instruction. These instructions guide the model to perform tasks such as adding, removing, replacing, or moving objects, as well as modifying attributes, without needing a separate text prompt. The system is trained on 1.55 million annotated edit pairs and uses a layer‑decoupled conditioning approach, achieving higher VLM rubric scores and PSNR on a benchmark that tests target selection among similar objects.

By Jinjing Zhao, Fangyun Wei, Yitong Wang, Xiuyu Wu, Yunuo Chen, Yang Yue, Sirui Zhang, Wenbo Wang, Hongyang Zhang, Dong Chen, Yan Lu, Chang Xu
arXiv AI
Oct 9

MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computing

MissMAC-Bench is a new benchmark for evaluating how multimodal affective computing systems handle missing modality data. It establishes fair, unified evaluation standards based on cross‑modal synergy, requiring models to perform without prior missing data during training and to handle both complete and incomplete inputs. The benchmark includes protocols for fixed and random missing patterns at dataset and instance levels, and experiments on three language models across four datasets demonstrate its effectiveness.

By Ronghao Lin, Honghao Lu, Ruixing Wu, Aolin Xiong, Qinggong Chu, Qiaolin He, Sijie Mai, Haifeng Hu
arXiv AI
Oct 9

RiCo: Neural Simulation of Rigid-Body Interactions via Local Contact Reasoning

RiCo is a neural simulation framework that models rigid‑body interactions by focusing on sparse neighborhoods of contact surface points. It integrates each point’s state with the geometry, motion, and physical properties of nearby surfaces, enabling local reasoning that preserves fine‑grained interaction details without modeling every pair of scene points. Experiments on the MOVi benchmark show that RiCo reduces 100‑frame position and orientation errors by 31‑35% and 38%, respectively, while achieving high contact fidelity and zero‑shot generalization to scenes with 270 objects.

By Ruixiang Ouyang, Guanren Qiao, Fansen Meng, Yueci Deng, Ruixing Jin, Kui Jia, Guiliang Liu
arXiv AI
Oct 9

ARC: A Reasoning Recipe for Robot Foundation Models

The paper introduces ARC, a reasoning recipe that enhances zero‑shot performance of robot foundation models (RFMs) without additional data or large‑scale training. ARC combines a reasoning trace grounded in the robot’s next action, an automatic labeling pipeline to generate these traces from existing demonstrations, and a strategy to adapt pretrained RFMs to use the traces for control. Applying ARC yields significant gains, including up to 50 percentage points on RoboLab-Reasoning-50 and an 82.2‑point improvement in task success for the π_{0.5} model on real robots.

By Gokul Puthumanaillam, Tao Sun, Elie Aljalbout, Moritz Reuss, Zhaoshuo Li, Fabio Ramos, Ankit Goyal, Jenai Xuning Yang
arXiv AI
Oct 9

SpaceFlow: Locally Controllable 3D Generation

SpaceFlow is a training‑free pipeline that enables locally controllable 3D generation from text and geometric primitives. Each primitive acts as a proxy for an object part and receives a local control level, allowing users to dictate strict adherence to input shapes or permit generative completion in different regions. The method enforces spatial constraints during structure generation and matches appearance to primitives, limiting cross‑part leakage and achieving state‑of‑the‑art prompt faithfulness and material accuracy.

By Neil De La Fuente, Joan Lafuente, Mukhammadali Sayfiddinov, Felicia Scharitzer, Marc Pollefeys, Ata Celen, Sayan Deb Sarkar, Elisabetta Fedele
arXiv AI
Oct 9

Predicting Alignment Generalization with Value Representations

The paper introduces the task of predicting alignment generalization, which measures how fine‑tuning a language model to follow a specific value affects its behavior across a broad set of other values. Through a large‑scale study of 66 values, the authors show that representations derived from model activations when applying values in context outperform textual description‑based methods, achieving a correlation of 0.45 versus 0.05. They further demonstrate that these representations can predict downstream robustness and help construct a taxonomy of LLM values based on empirical generalization dynamics.

By Andy Liu, Mehar Bhatia, Karolina Stanczak, Mona Diab, Vered Shwartz, Daniel Fried
arXiv AI
Oct 9

Machine learning and information theory concepts towards an AI Mathematician

The paper argues that current AI excels at intuitive, system‑1 tasks but falls short on system‑2 reasoning needed for mathematics. It proposes using information theory to define what makes a mathematical statement interesting, aiming to guide the creation of an AI mathematician that can generate novel conjectures rather than merely prove existing theorems. The central hypothesis is that a concise, descriptive set of theorems can efficiently capture the space of all provable statements by minimizing description length while remaining close to many derivable results.

By Yoshua Bengio, Esmeralda S. Whitammer
arXiv AI
Oct 9

Neural Architecture Discovery via Autonomous Evolution

The paper introduces ASI-Arch, an AI-driven system that autonomously conducts neural architecture research through a closed-loop cycle of research, experiment, analyze, and update. In experiments focused on linear attention, ASI-Arch performed 1,773 iterations and identified 105 state‑of‑the‑art architectures, with its top design surpassing DeltaNet by nearly three times the improvement achieved by Mamba2. The study also dissects how various components of the framework contribute to success in this demanding research context.

By Weixian Xu, Yixiu Liu, Yang Nan, Lyumanshan Ye, Xiangkun Hu, Zhen Qin, Pengfei Liu