Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI
Oct 2

Improving Math Reasoning through Value-guided Informative Search

The paper introduces APIVIS, a training-time framework that integrates finite-budget Gumbel search into reinforcement learning with verifiable rewards (RLVR) for mathematical reasoning. APIVIS combines direct and searched responses within each rollout group, uses selective supervision on search-improved tokens, and applies value-guided selection to improve verifier rewards at each searched state. Experiments on standard mathematical reasoning benchmarks and various model scales show significant performance gains over existing search-based methods.

By Shaohuai Liu, Yuning Wu, Haoran Liu, Enzo Jia, Devin Chen, Kai Wei
arXiv AI
Oct 2

CortexBridge: Cortical Alignment of EEG Montages for Foundation Models

CortexBridge is a lightweight adapter that aligns arbitrary EEG electrode montages into a shared cortical latent space by combining EEG features with electrode and atlas coordinates. When evaluated on five BCI datasets from MOABB with three frozen foundation models, it improves performance in 13 of 15 tasks, achieving average balanced‑accuracy gains of 0.80% (EEGPT), 0.70% (LaBraM), and 3.26% (CBraMod), with a maximum 13.02% improvement on a 12‑class SSVEP task. Visualizations show task‑dependent spatial patterns, such as a concentrated representation in the Yeo Visual network for SSVEP versus auditory P300.

By Jiazhen Hong, Xiaotian Zhou, Zihao Ding, Kailong Wang, Yu Wu
arXiv AI
Oct 2

ProtoFlow: Prototype-Guided Flow Matching for Multivariate Time Series Forecasting

ProtoFlow is a new multivariate time series forecasting framework that combines vector‑quantized autoencoding with prototype‑guided flow matching. It maps sequences into a discrete latent space, constructs a structured prior from the learned VQ codebook, and trains a DiT‑based rectified flow to transport samples from this prior to future latent representations conditioned on past observations. By replacing generic Gaussian noise with a learned prototype prior, ProtoFlow eliminates autoregressive rollout mismatch and achieves faster training convergence while delivering superior forecasting performance on benchmark datasets.

By Shibo Feng, Wanjin Feng, Yang Qiu, Deheng Ye, Peilin Zhao, Chunyan Miao
arXiv AI
Oct 2

TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safety

The paper introduces TRACE, a token‑level objective designed to reduce multi‑turn safety risks in large language models. TRACE assigns each token a weight based on the discounted return of a refusal‑attributable advantage, comparing a frozen reference model with a refusal‑ablated copy to credit early tokens for later refusal evidence. Evaluated across five open‑weight models and seven multi‑turn attacks, TRACE achieves the lowest attack success rate in all 35 model‑attack pairs while maintaining model utility within 1.23 points on MMLU and HellaSwag.

By Fengpeng Li, Kemou Li, Qizhou Wang, Haiwei Wu, Jiantao Zhou, Di Wang
arXiv AI
Oct 2

PRISM: A Category-Theoretic Framework for Measuring and Refining Multimodal Analogies

PRISM is a modality‑agnostic, category‑theoretic framework that measures and refines multimodal analogies by representing them as explicit relational mappings. It introduces a pullback score to quantify relational alignment and an iterative refinement loop that uses this score as feedback to improve generated images. On the AnaloBench benchmark, PRISM’s pullback score alone achieves 82.5% accuracy, and human evaluations show a 57.65% preference for refined outputs, though refinement may sometimes favor visually crowded compositions.

By Mirella Zeisler, Ojas Shirekar, Mircea Lic\v{a}, Chirag Raman
arXiv AI
Oct 2

DRelay: Global Draft Context for Prefix-Aware Parallel Speculative Decoding Repair

DRelay introduces a global draft context mechanism to improve prefix-aware parallel speculative decoding for large language models. By using a global reader to extract predictive information across the entire draft block and a causal selector to repair early token selection errors, DRelay extends the accepted prefix length and enhances decoding performance. Experiments on eight benchmarks show consistent gains over existing methods such as DFlash, Domino, and DSpark, with notable speedup improvements in SGLang serving.

By Zhuoyu Wang, Junnan Huang, Xinyu Chen
arXiv AI
Oct 2

OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation

OpenMTB‑Audit is an open‑source benchmark that tests large language models on 500 synthetic non‑small cell lung cancer cases, covering five adversarial error categories and four safety labels: Supported, Partially Supported, Unsupported, and Insufficient Information. The study found that all eight tested LLMs over‑refused Partially Supported recommendations, collapsing labels to achieve high safety scores. A deterministic seven‑module framework, MTB‑AuditAgent, was introduced to reduce over‑refusal to 6.7% and reach 91.2% accuracy, while an oncologist annotation study highlighted disagreement around the boundary between information sufficiency and treatment optimization.

By Negin Ashrafi, Jia Luo, Stacey M. Frumm, Roxana Daneshjou
arXiv AI
Oct 2

RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations

RealCompanion is a benchmark that evaluates an AI companion’s ability to understand a human over long, real-world conversations. It consists of ten real relationships with 27,218 messages spanning up to 120 days, along with derived files such as a profile, persona, chat ground truth, and a question set that cites the relevant messages. The study finds that past context is rarely needed, memory detection is challenging, and agent systems vary widely in cost while achieving similar persona reconstruction.

By Arman Behnam, Sunglyoung Kim, Liangwei Yang
arXiv AI
Oct 2

Measuring the Stability Assumption Behind Action Chunking

The paper investigates how small action errors evolve when using action chunking in behavioural cloning. By injecting errors at each state and observing their growth under open‑loop (no replanning) and closed‑loop (replanning) regimes, the authors classify states as contracting, expanding, or unresolved. Across twelve manipulation tasks, they find that stable states are rare, error amplification is common, and that short‑horizon fitting can overestimate long‑horizon propagation. Predictors trained on camera and proprioceptive data can recover open‑loop stability but only partially capture closed‑loop dynamics, indicating that standard imitation learning does not reliably produce policies that contract errors when perturbed.

By Aryan Goyal
arXiv AI
Oct 2

LineupRL: Verifiable Reinforcement Learning for Time Series Captioning via Caption-to-Series Identification

LineupRL introduces a reinforcement learning framework with verifiable rewards for time series captioning, using a frozen large language model to identify the correct time series from a set of distractors based on a generated caption. This approach bypasses the limitations of supervised fine‑tuning and traditional RL rewards that poorly transfer to open‑ended time series generation. Experiments on two captioning benchmarks, as well as forecasting and reconstruction tasks, show that LineupRL outperforms both SFT and RL baselines across all metrics, and its trained 3B vision‑language model surpasses a 72B model distilled from SFT captions. The method also demonstrates resistance to reward hacking and produces captions that accurately trace trends and name key values.

By Haochen Zhang, Laura Yao, Zachary Plotkin, Gengwei Zhang, Tianlong Chen
arXiv AI
Oct 2

AVSD-Scenes: A Dataset for Audio-Visual Description of Urban Scenes

AVSD-Scenes is a new dataset of 12,291 audio‑visual scene descriptions for urban environments, built from the TAU Urban Audio‑Visual Scenes dataset. The descriptions are generated by first creating modality‑specific text with Qwen2‑Audio‑7B and Qwen2.5‑VL‑7B, then merging them with large language models (Qwen3‑14B, Mistral‑Small‑3.2‑24B‑Instruct‑2506, Gemma‑3‑27B‑it) to produce multimodal narratives that combine auditory and visual cues. Benchmarks show that these multimodal descriptions improve semantic alignment, cross‑modal retrieval, and scene classification accuracy (up to 95.4%) while remaining discriminative even without explicit scene labels.

By Dhanunjaya Varma Devalraju, Arshdeep Singh, Mark D. Plumbley
arXiv AI
Oct 2

Counting Moves, Weighing Voices: Bayesian Dialectical Argumentation for Calibrated Multi-LLM Councils under Persistent Adversaries

The paper introduces Bayesian Dialectical Argumentation (BDA), a method for aggregating answers from multiple large language models (LLMs) in a council setting. BDA treats each LLM’s typed moves—proposals, challenges, and concessions—as evidence in a classical annotator model, estimating per-agent reliability even when some agents are persistently unreliable. By weighting evidence according to these inferred reliabilities, BDA produces calibrated posterior probabilities for candidate answers and can invert unreliable agents instead of merely outvoting them, achieving superior calibration and robustness on both binary and multi-class benchmarks without extra LLM calls.

By Ionel Eduard Stan, Paolo Napoletano
arXiv AI
Oct 2

SPHERE: Adaptive VR Indoor Scene Generation via LLM-Enhanced Spatial Preference Learning and Human-in-the-Loop RL

SPHERE is an adaptive VR indoor scene generation framework that turns isolated 3D synthesis into continuous human‑AI co‑creation. It learns persistent spatial preferences from multimodal user interactions, abstracts these into hierarchical constraints for geometric resilience, and employs a human‑in‑the‑loop reinforcement learning loop to refine retrieval policies. A mixed‑design study with 42 participants and offline ablation show that SPHERE reduces corrective edits and physical effort while avoiding bias toward shallow object‑level traits, producing geometrically resilient, profile‑aligned layouts.

By Hyeonmin Lee, Zheng Wei, Kyungmin Kwon, Jumin Seo, Jiwon Park, Hayoung Oh
arXiv AI
Oct 2

Mimir: Physics-Grounded LLM Agents for Long-Horizon Irrigation Control

Mimir is a physics‑grounded large language model agent designed for long‑horizon irrigation control. It operates on two timescales: a fast scale that uses a structured physical interface and deterministic simulator to validate and refine LLM proposals before execution, and a slow scale that consolidates recurrent failure patterns into persistent contextual principles. Across multiple sites, crops, and years, Mimir achieves the lowest aggregate control cost and reduces irrigation usage by about 51% compared to historical schedules, while ablation studies confirm the importance of forward simulation, verified revision, and persistent context.

By Yimeng Liu, Mi Zhang, Younsuk Dong, Zhichao Cao
arXiv AI
Oct 2

VISTA: A Visual Harness for Reasoning in an Interactive World

The paper introduces VISTA, a visual harness that equips a general-purpose multimodal model with long‑horizon vision and a lossless visual memory. VISTA enables the model to directly perceive and actively retrieve past observations, allowing it to reorganize visual input during reasoning. On the ARC‑AGI‑3 benchmark, VISTA boosts Claude Opus 5.0’s Relative Human Action Efficiency from 40.68 to a perfect 100.00, completing all 25 public games with 57.4% fewer actions than first‑time human participants, and it also outperforms baselines on three additional visual game and puzzle benchmarks.

By Qiushi Han, Keya Hu, Linlu Qiu, Cathy Wu, Kaiming He
arXiv AI
Oct 2

New Snake-in-the-Box Records via Snakepit Surgery and Learned Construction

The paper reports a new record for the snake-in-the-box problem, presenting a length‑191 snake in the 9‑dimensional hypercube, surpassing the previous 190‑length record that had stood for 14 years. It also establishes new lower bounds for dimensions 10 through 13. The authors introduce a novel search technique called snakepits, which expands the search space by considering collections of disjoint snakes, and propose a new benchmark, Snakepit‑in‑the‑Box, that evaluates maximal edge counts across multiple components. Additionally, they present Beam Anchor, a learned construction algorithm that generates 100 inequivalent length‑190 snakes in dimension 9.

By Paul Orland, Lucas Fagan, Michele Tarquini, Davide Passaro, Maksymilian Manko, Elli Heyes, Angus Gruen, Giorgi Butbaia, Justin Tan, Sergei Gukov
arXiv AI
Oct 2

STATERA: Hidden Mass Estimation via Zero-Shot Sim-to-Real Kinematics using Frozen Temporal Tubelets

STATERA is a method that adapts a pretrained video backbone with mostly frozen weights and a lightweight temporal tubelet mixer to estimate the center-of-mass (CoM) of opaque, asymmetric rigid bodies from short monocular videos. It introduces the HiddenMass Benchmark, consisting of 50K simulated MuJoCo trajectories and a 63-sequence real-world test set with calibrated CoM ground truth. In simulation, STATERA reduces normalized CoM error from 41.7% to 25.2%, and in zero-shot sim-to-real transfer, its phase‑aware variant consistently predicts movement toward the true hidden offset, improving physics capture from 2.6% to 41.0%.

By Animesh Varma
arXiv AI
Oct 2

SW-KAN: Kolmogorov-Arnold Networks with Stieltjes-Wigert q-Orthogonal Polynomials

The paper introduces SW-KAN, a Kolmogorov‑Arnold Network that replaces traditional B‑spline activations with Stieltjes‑Wigert q‑orthogonal polynomials defined on the semi‑infinite domain (0, ∞). It addresses the domain mismatch between unbounded inputs and bounded polynomial bases by applying a smooth exponential‑of‑tanh mapping, and uses a numerically stable three‑term recurrence to evaluate polynomial expansions efficiently. Experiments on image classification and continuous function approximation show that SW‑KAN achieves better accuracy‑efficiency trade‑offs than existing polynomial KANs, especially in resource‑constrained scenarios with limited data or feature dimensionality.

By Amirhosein Azarpour, Seyyed Moein Kazemi
arXiv AI
Oct 2

Spatial Lifting for Dense Prediction

Spatial Lifting (SL) is a new method for dense prediction that lifts 2D inputs into a higher-dimensional space and processes them with networks suited to that dimension, such as a 3D U‑Net. This dimensionality lifting surprisingly yields competitive performance while cutting inference costs and dramatically reducing model parameters. The SL framework naturally produces structured outputs along the lifted dimension, enabling dense supervision during training and single‑forward‑pass self‑consistency‑based quality and uncertainty estimation at test time.

By Mingzhi Xu, Tao Zhou, Yong Li, Yizhe Zhang
arXiv AI
Oct 2

Decoding the Disaster: Multi-Task Geospatial Reasoning with Vision-Language Models and Crowdsourced Imagery for Disaster Mapping

The paper introduces GRDisaster, a multi-task geospatial reasoning framework that leverages vision‑language models to interpret, geolocalize, and assess damage in crowdsourced disaster imagery. It builds on a new benchmark dataset of 26,340 images from PhotoMappers, linking volunteer geographic information, street‑view imagery, and remote sensing data across multiple disaster events from 2018 to 2024. GRDisaster combines deterministic and probabilistic cross‑view geolocalization with multi‑view fusion, and introduces spatial reasoning indicators to validate cross‑view matches and quantify disaster severity using expert‑verified annotations.

By Wenping Yin, Fabian Desuer, Ziqi Liu, Naixia Mou, Weijia Li, Pedram Ghamisi, Xiao Xiang Zhu, Hao Li