Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI
Sep 28

Selective Off-Policy Reference Tuning with Plan Guidance

Selective Off-Policy Reference Tuning with Plan Guidance (SORT) is a reinforcement learning approach that improves reasoning by adding a repair update for failed rollouts. It derives a plan from a reference solution, compares token probabilities with and without that plan, and gives higher weight to tokens that become more predictable under plan conditioning. This method turns all-wrong prompts into selective, structure-aware learning signals, yielding improvements over GRPO and guidance baselines across three backbones and eight reasoning benchmarks, especially on weaker models.

By Anh Duc, Tien-Phat Nguyen, Thien Huu Nguyen, Linh Ngo Van, Trung Le
arXiv AI
Sep 28

Nice Fold or Hero Call: Learning Budget-Efficient Thinking under Policy-Dependent Solvability

The paper introduces Budget‑Efficient Thinking (BET), a two‑stage framework that treats adaptive reasoning as a computational investment, aligning solve‑or‑fold decisions with expected return rather than perceived difficulty. BET learns three distinct behaviors: concise short solves for easy queries, early abstention (nice fold) when further reasoning is unlikely to pay off, and allocating sufficient compute (hero call) for hard‑but‑solvable questions. Experiments on seven benchmarks with three base models show BET cuts reasoning tokens by 54% while boosting accuracy by up to 3.2%, and it transfers effectively to scientific QA and logical reasoning tasks.

By Zhaomeng Zhou, Lan Zhang, Junyang Wang, Mu Yuan, Songlin Liu, Tingzhao Li, Yiqing Hu, Yumeng Zhao
arXiv AI
Sep 28

Energy Efficiency of Locally Deployed LLMs: A Preliminary Quantitative GPU Power Benchmark on Consumer Hardware

The paper reports a reproducible GPU power benchmark for 18 open‑source LLMs (0.5B–7B parameters) run on a single consumer RTX 4060ti GPU using the Ollama inference engine. Energy metrics such as mean/peak power, total energy per prompt, energy per output token, and throughput were measured, revealing that model architecture and quantization strategy, rather than parameter count alone, drive energy efficiency. The most efficient models were qwen2.5:0.5b and tinyllama:1.1b, while the 7B‑Mistral model consumed up to 8.6× more energy per token, and qwen3.5:0.8b(on) showed unusually high per‑prompt energy due to extended internal reasoning.

By Philipp M. Z\"ahl, Elja Dalipaj, Anika Hennig, Timon Bayer
arXiv AI
Sep 28

LEAD: An EEG Foundation Model for Alzheimer's Disease Detection

LEAD is a gated temporal‑spatial Transformer foundation model designed for EEG‑based Alzheimer's disease detection. It was trained on the world’s largest EEG‑AD corpus of 2,238 subjects and uses a subject‑regularized strategy and medical contrastive learning across 13 datasets. LEAD outperforms existing methods on five downstream AD datasets, achieving the best average ranking across 20 evaluations.

By Yihe Wang, Nan Huang, Nadia Mammone, Marco Cecchi, Xiang Zhang
arXiv AI
Sep 28

LapDDPM: Spectral Perturbation Diffusion for Robust Single-Cell Manifold Generation

LapDDPM is a conditional Graph Diffusion Probabilistic Model that generates high‑fidelity, biologically plausible single‑cell RNA sequencing data. It incorporates graph‑based inductive biases and a spectral adversarial perturbation mechanism to enforce robustness against structural noise, effectively acting as a Distributionally Robust Optimization framework. The model extends to spatial transcriptomics and multi‑modal data, and experimental results on datasets such as PBMC3K, Dentate Gyrus, HLCA, Visium, and 10x Multiome show it outperforms state‑of‑the‑art baselines in distribution matching, manifold preservation, and downstream utility.

By Lorenzo Bini, Stephane Marchand-Maillet
arXiv AI
Sep 28

AUWave: A Data-Driven Model for Reconstructing Significant Wave Heights Using Sparse Observations

AUWave is a hybrid deep‑learning framework that reconstructs high‑resolution regional significant wave height (SWH) fields from sparse buoy observations. By combining a station‑wise encoder with a multi‑scale U‑Net enhanced by self‑attention, it outperforms a baseline model, especially when more than one buoy is available, and identifies critical anchor stations through buoy ablation studies. Cross‑basin tests in the Atlantic and Pacific demonstrate the model’s robustness and portability, suggesting its applicability for operational ocean monitoring and data assimilation.

By Hongyuan Shi, Yilin Zhai, Ping Dong, Zaijin You, Chao Zhan, Qing Wang
arXiv AI
Sep 28

BabelCoder: Agentic Code Translation with Specification Alignment

BabelCoder is an agentic framework for automatic code translation that splits the task into specialized agents for translation, testing, and refinement. Each agent focuses on a specific aspect—generating code, validating correctness, or repairing errors—allowing collaborative improvement of translation quality. Evaluated on four benchmark datasets, BabelCoder outperforms four state‑of‑the‑art baselines, achieving an average accuracy of 94.16% and surpassing existing methods in 94% of cases.

By Fazle Rabbi, Soumit Kanti Saha, Tri Minh Triet Pham, Song Wang, Jinqiu Yang
arXiv AI
Sep 28

Consist-Retinex: One-Step Noise-Emphasized Consistency Training Accelerates High-Quality Retinex Enhancement

Consist‑Retinex introduces a one‑step noise‑emphasized consistency training framework for Retinex‑based low‑light image enhancement. It first decomposes images into reflectance and illumination maps using a Retinex Transformer Decomposition Network, then trains two conditional consistency models with a dual objective that blends trajectory consistency and ground‑truth alignment. The method employs adaptive noise‑emphasized fixed‑point sampling to focus supervision near the inference endpoint, achieving state‑of‑the‑art VE‑LOL‑L scores on paired and unpaired low‑light benchmarks while reducing sampling and training costs.

By Jian Xu, Wei Chen, Shigui Li, Delu Zeng, John Paisley, Qibin Zhao
arXiv AI
Sep 28

Geometric-Photometric Event-based 3D Gaussian Ray Tracing

The paper introduces GPERT, a framework that separates event-based 3D Gaussian Splatting into two rendering branches: event-by-event geometry rendering and snapshot-based radiance rendering. By employing ray-tracing and warped event images, GPERT balances accuracy and temporal resolution, achieving state‑of‑the‑art results on real‑world datasets and competitive performance on synthetic data. The method operates without pretrained models or COLMAP initialization, offers flexible event selection, and produces sharp reconstructions of scene edges with rapid training.

By Kai Kohyama, Yoshimitsu Aoki, Guillermo Gallego, Shintaro Shiba
arXiv AI
Sep 28

Dynamic Welfare-Maximizing Pooled Testing

The paper studies a budget‑constrained welfare problem for pooled testing, where agents have heterogeneous utilities and independent probabilities of being healthy. It proves that an optimal dynamic testing policy can achieve at most twice the welfare of the best static overlapping allocation, regardless of population, budget, or pool‑size limit. The authors also identify cases where adaptivity offers no benefit, show that re‑pooling after positive tests is necessary for strict gains, and provide approximation guarantees for greedy algorithms.

By Edwin Lock, Nicholas Lopez, Francisco Marmolejo-Coss\'io, Jose Roberto Tello Ayala, David C. Parkes
arXiv AI
Sep 28

RAPTOR: Ridge-Adaptive Logistic Probes

RAPTOR (Ridge-Adaptive Logistic Probe) is a lightweight, L2‑regularized logistic probe designed to extract accurate, directionally stable concept vectors from frozen large language model (LLM) layer representations. By tuning the ridge penalty on validation data, RAPTOR produces concept vectors from normalized weights that match or surpass strong baselines in accuracy, maintain stability under ablation, and require significantly less training cost. The authors also provide a mechanistic analysis using the Convex Gaussian Min‑max Theorem to explain how penalty strength influences probe accuracy and stability in a high‑dimensional few‑shot setting, with predictions that align with empirical observations on real LLM embeddings.

By Ziqi Gao, Yaotian Zhu, Qingcheng Zeng, Xu Zhao, Ziqing Wang, Feng Ruan, Kaize Ding
arXiv AI
Sep 28

HuPER: A Human-Inspired Framework for Phonetic Perception

HuPER is a human-inspired framework that models phonetic perception as adaptive inference over acoustic‑phonetics evidence and linguistic knowledge. Using only 100 hours of training data, it achieves state‑of‑the‑art phonetic error rates on five English benchmarks and demonstrates strong zero‑shot transfer to 95 unseen languages. It uniquely enables adaptive, multi‑path phonetic perception across diverse acoustic conditions, and all training data, models, and code are open‑sourced.

By Chenxu Guo, Jiachen Lian, Yisi Liu, Baihe Huang, Shriyaa Narayanan, Bixing Wu, Zoe Ezzes, Jet Vonk, Zachary Miller, Cheol Jun Cho, Maria Gorno-Tempini, Gopala Anumanchipalli
arXiv AI
Sep 28

VLANeXt Family: A Systematic Study of VLA Models from Core Recipes to Emerging Paradigms

The paper presents VLANeXt, a Vision‑Language‑Action (VLA) model built from a unified framework that systematically analyzes design choices across foundational components, perception essentials, and action modeling. By dissecting these dimensions, the authors identify 12 key findings that form a practical recipe for strong VLA models, and demonstrate that VLANeXt outperforms state‑of‑the‑art methods on LIBERO and LIBERO‑plus benchmarks while also excelling in real‑world experiments. The study further extends VLANeXt into a family of variants—scaled models, latent‑action pretraining, latent predictive representation learning, and world action modeling—showing that the core recipe remains effective across different scales and emerging paradigms.

By Xiao-Ming Wu, Kang Liao, Yihang Luo, Bin Fan, Jian-Jian Jiang, Runze Yang, Zhonghua Wu, Wei-Shi Zheng, Chen Change Loy
arXiv AI
Sep 28

VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation

VLAA-GUI is a modular framework for autonomous GUI agents that addresses early stopping and repetitive loops by integrating three core components: a Completeness Verifier, a Loop Breaker, and an on-demand Search Agent. The framework also includes optional Coding and Grounding Agents for specialized tasks. Evaluations on five backbones across Linux and Windows benchmarks show strong performance, with some models surpassing human results and the Loop Breaker significantly reducing wasted steps.

By Qijun Han, Haoqin Tu, Zijun Wang, Haoyue Dai, Yiyang Zhou, Nancy Lau, Alvaro A. Cardenas, Yuhui Xu, Ran Xu, Caiming Xiong, Zeyu Zheng, Huaxiu Yao, Yuyin Zhou, Cihang Xie
arXiv AI
Sep 28

Human-1 by Josh Talks: A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations

The paper introduces Human-1, the first open and reproducible full‑duplex spoken dialogue system for Hindi. It adapts the Moshi architecture with a custom Hindi tokenizer and trains on 26,000 hours of real spontaneous conversations from 14,695 speakers, enabling the model to learn natural turn‑taking, interruptions, overlaps, and backchannels. A two‑stage training process—large‑scale pre‑training followed by fine‑tuning on 1,000 hours—yields a system that, according to both automatic metrics and human judgments, generates natural and meaningful full‑duplex conversational behavior in Hindi.

By Bhaskar Singh, Manas Dhir, Shobhit Banga, Pranav Sharma
arXiv AI
Sep 28

Attention Sinks and Outliers in Attention Residuals

The paper introduces OASIS, a method designed to stabilize dual‑normalized attention‑residual architectures by employing explicit null routing and token‑to‑depth null coupling. OASIS mitigates attention sinks and activation outliers, improving low‑bit quantization performance across several language‑model backbones. Empirical results show significant reductions in attention norms and perplexity, with notable gains on long‑context benchmarks.

By Haozheng Luo, Haoran Dai, Jingyuan Huang, Shaoyang Zhang, Xi Chen, Eric Hanchen Jiang, Yijiang Li, Chenghao Qiu, Chenwei Xu, Zhenyu Pan, Haotian Zhang, Binghui Wang, Yan Chen
arXiv AI
Sep 28

Hide-and-Seek in Trajectories: Discovering Failure Signals for VLA Runtime Monitoring

The paper introduces Hide-and-Seek, a framework for detecting failures in Vision‑Language‑Action (VLA) models during robot execution. It treats failure detection as a coarsely supervised learning problem, using inter‑trajectory and intra‑trajectory contrastive objectives to localize failure‑indicative actions without step‑level annotations. Experiments on LIBERO, VLABench, and a real‑world robotic platform show that Hide‑and‑Seek achieves state‑of‑the‑art multi‑task failure detection performance across several VLA policies.

By Seongheon Park, Wendi Li, Changdae Oh, Samuel Yeh, Zsolt Kira, Michael Hagenow, Sharon Li
arXiv AI
Sep 28

Efficient Safety Benchmarking via Item Response Theory

The paper demonstrates that Item Response Theory (IRT) can uncover meaningful structure in safety benchmarks for language models, allowing adaptive item selection to approximate full benchmark rankings with Spearman’s ρ > 0.90 while cutting evaluation costs by at least 80% and up to 99.9% on some suites. It also proposes a static method to extract a small, informative subset of items that can be reused across models, achieving 80–99.8% cost savings. These findings show that psychometric techniques can make safety evaluation more efficient without sacrificing ranking accuracy.

By Fabio Spagliardi, M\'irian Silva, Ayan Datta, Aiden Zhou, Vamshi Bonagiri, Diogo Cruz