Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Computation and Language
Oct 9

Fact over Fiction: Detection of Pathological Hallucinations in Sinhala-to-English Neural Machine Translation

The paper presents a reference‑free framework for detecting hallucinations in Sinhala‑to‑English neural machine translation. It introduces a 45,000‑sample synthetic dataset created via five linguistically motivated corruption strategies and a semantic rescue mechanism that uses character‑level similarity. Using a fine‑tuned mDeBERTa‑v3 model, the authors achieve a token‑level F1 of 0.841 and demonstrate that the detector relies on the Sinhala source, not on surface artefacts, with an AUROC of 0.970 on a source‑disjoint test set. The study also benchmarks eight NMT systems, showing that hallucination rates vary by an order of magnitude across architectures.

By Navam Obeysekara, Nevidu Jayatilleke
arXiv Machine Learning
Oct 9

SciTBERT: A family of chronologically consistent language models for scientific and technological language processing

SciTBERT is a family of BERT-derived language models trained on chronologically consistent corpora of scientific papers, patents, and educational web text, with training cutoffs spanning 2013 to 2025. The models are further post‑trained using paper and patent citations to create the SciTBERT‑CI family, which better captures temporal dynamics. Experiments show that these chronologically consistent encoders outperform earlier domain‑specific models, especially when training data is limited by early year restrictions, and they perform well on the new PatRepEval benchmark of patent‑related tasks at the science‑technology interface.

By Thomas Gebhart, Russell J. Funk
arXiv Machine Learning
Oct 9

Measuring and Mitigating Solution Mode Collapse in RLVR

The paper introduces ModeBench, a benchmark for multi-solution tasks that evaluates both correctness and the diversity of solutions found by a language model. It shows that reinforcement learning with verifiable rewards (RLVR) tends to collapse solution modes, concentrating probability on fewer correct answers even while accuracy remains high. To counter this, the authors propose Re:Max, a replay-based method that stores one verified example per discovered mode and trains uniformly on these stored modes, improving both success rates and solution diversity across various model scales and RL objectives.

By Liv G. d'Aliberti, Marwa Abdulhai, Sofiia Druchyna, Peter Henderson, Manoel Horta Ribeiro
arXiv AI
Oct 9

REMORY: Learning Residual Memory for Context Compaction

REMORY is a neural memory network that augments a textual summary of an agent’s history with a small set of soft memory tokens. These tokens, conditioned on the summary, enable a frozen large language model to approximate the continuation it would generate using the full history, effectively acting as a residual connection along the sequence dimension. Experiments on SummHay and long‑horizon agent benchmarks show that REMORY improves source attribution, maintains insight coverage, and achieves near full‑context performance while using only 5.2% of the input positions, while also reducing repeated tool outputs and errors on BrowseComp and Terminal‑Bench 2.1.

By Hanchen Xia, Baoyou Chen, Yutang Ge, Naihao Deng, Senqiao Yang, Zilong Dong, Weihao Yuan, Siyu Zhu
arXiv AI
Oct 9

Zatom-2: Multitask Pretraining on Atomistic Data for Generative Modeling across Domains

Zatom-2 is a multitask generative model for atomistic data that has been pretrained on about five million structures from the OMol25 and OMat24 datasets. It uses a multiscale Transformer with conditional flow matching to support tasks such as generation, structure prediction, and energy/force prediction for both molecules and materials. The model outperforms its predecessor, Zatom-1, on molecular distribution fidelity and benchmark generation tasks, and improves protein backbone designability from 67.8% to 74.8% after finetuning on 2,000 protein domains.

By Miruna Cretu, Alex Abrudan, Antonia Panescu, Tynan Perez, Rishabh Anand, N. Benjamin Erichson, Michael W. Mahoney, Samuel Blau, Joseph Jacobson, Rafael G\'omez-Bombarelli, Rex Ying, Tuomas Knowles, Pietro Li\`o, Alex Morehead
arXiv Machine Learning
Oct 9

Bilevel optimization for data-driven learning of Koopman embeddings using kernel-based autoencoders

The paper introduces EDMD-kDL, a kernel-based method that learns finite-dimensional Koopman embeddings directly from data using bilevel optimization and collocation techniques. It combines kernel dictionary learning with extended dynamic mode decomposition to produce Koopman approximations without pre-specifying a dictionary. Experiments on sea-surface-temperature forecasting and video data show that EDMD‑kDL matches or surpasses ANN-based approaches while remaining scalable to large datasets.

By Joel-Pascal Ntwali N'konzi, Feliks N\"{u}ske, Stefan Klus
arXiv AI
Oct 9

SpikeSSL: A Universal Spike Inference Framework with Dynamics-Informed State-Space Layers

SpikeSSL is a universal spike inference framework that uses bidirectional IIR state‑space layers inspired by calcium dynamics as its temporal backbone. It incorporates a multi‑modal conditioning encoder to adapt to different calcium indicators, sampling rates, and trace statistics, and provides per‑frame uncertainty estimates. On a benchmark of 33 public datasets, SpikeSSL achieves state‑of‑the‑art performance both in‑domain and in zero‑shot leave‑one‑indicator‑out settings, and a biophysical simulation pipeline further improves cross‑indicator generalization.

By Chenghao Yue, Siming Xing, Shuran Liu, Angran Li, Yuanlong Zhang
arXiv Machine Learning
Oct 9

Directly Optimizing Mean Demographic Parity for Nonlinear Regression

The paper introduces DPVar, a fairness metric that measures the variance of the conditional mean prediction to directly optimize mean demographic parity in nonlinear regression. It presents two solvers—FBO, using a closed‑form hypergradient, and ITD, which differentiates through the conditional‑mean estimator updates—to handle the resulting functional bilevel problem. Experiments on 21 tabular datasets and the Communities & Crime data show that these methods achieve competitive or superior accuracy‑fairness trade‑offs compared to existing approaches.

By Ieva Petrulionyte, Julien Mairal, Michael Arbel
arXiv AI
Oct 9

Mine Odyssey: Benchmarking Spatial Agentic Intelligence in the Wild

Mine Odyssey is a new benchmark that tests agentic spatial intelligence by having agents navigate Minecraft reconstructions of real-world locations. The benchmark includes 180 tasks across 30 sites in 20 countries, featuring 20 outdoor and 10 indoor settings with varied scales, layouts, and terrains. Each task gives a natural‑language instruction to visit specific waypoints in order, requiring agents to find routes, open doors, and manage navigation errors.

By Yuxuan Cao, Junlong Li, Hao Li, Junxian He
arXiv AI
Oct 9

GeoReform: Reflective Formalization Evolution for Multimodal Geometry Problem Solving

GeoReform is a reflective formalization evolution framework designed to improve multimodal geometry problem solving by treating formalization as an optimizable policy rather than a fixed parser. It iteratively runs the reasoning pipeline, collects failures, diagnoses representation defects, and mutates the policy to better select, ground, group, and present geometric entities, relations, constraints, and targets. On the Geometry3K benchmark, GeoReform raises Qwen3VL-2B accuracy from 42.0% to 56.0%, demonstrating that effective formalization is crucial for enhancing multimodal geometry reasoning.

By Jialu Wang, Ruichen Zhang, Xiaoou Liu, Hua Wei, Tianlong Chen
arXiv AI
Oct 9

Empty Commitments: When Agents Promise What They Cannot Deliver

The paper introduces the concept of "empty commitments," promises made by AI agents that cannot be fulfilled due to the absence of necessary tools or runtime capabilities. It defines three failure types, proposes a detection checker, and evaluates it on 400 human-labeled examples and a benchmark of 293 follow‑up requests. The study shows that many open‑weight models frequently issue such empty commitments, and that simple interventions—informing the model of its runtime, adding a capability card, or running a checker loop—can significantly reduce these failures.

By Jiaqi Tang, Bingyu Shen, Lan Wei, Qing Lu, Bethel Ololade, Danny Galvis, Miles Q. Li, Bin Hu, Boyang Li
arXiv Machine Learning
Oct 9

RH-Detect: A Unified Benchmark for Reward Hacking Detection

RH-Detect is a unified benchmark for detecting reward hacking in language models, aggregating 92,761 examples from eleven public datasets into a common schema with six behavior categories. It evaluates six off‑the‑shelf models on 5,021 open‑ended evaluation units, achieving a top AUROC of 0.962 and over 93% accuracy, though performance drops notably on multi‑turn tool‑use datasets. The study also shows that input format changes and single‑token fine‑tuning can influence detector performance, revealing gaps in deployment‑time monitoring.

By Junwei Quan, Evgenii Opryshko, Rohan Subramani, Igor Gilitschenski
arXiv Machine Learning
Oct 9

Beyond Distributional Fidelity: Causal-Penalized Diffusion for Synthetic Tabular Data

The paper introduces a causal‑penalized diffusion model for generating synthetic tabular data, arguing that high statistical fidelity does not guarantee accurate causal inference. It defines causal fidelity as the discrepancy between inferential distributions from real and synthetic data and proposes a training framework that adds a causal discrepancy penalty to the generative objective. Experiments on treatment‑effect simulations and benchmark datasets demonstrate that the method can improve causal fidelity while maintaining competitive statistical fidelity.

By Lan Tao, Yongxian He, Shirong Xu, Yidong Ouyang, Guang Cheng
arXiv Machine Learning
Oct 9

Perception Test 2026: Challenge Summary and Extension to City-scale Audio-Visual Reasoning

The fourth Perception Test challenge was held as a workshop at ECCV 2026 in Malmö, Sweden, focusing on spatial intelligence. It included four tracks: unified multiple-choice videoQA, grounded videoQA from the original benchmark, and two new city-scale walking-tour tracks—KilometerAudio and KilometerVision. The report details the new benchmarks and summarizes the winning solutions, noting that while expensive agentic pipelines succeeded in the city-scale tracks, standalone multimodal models still struggle with complex spatial and multimodal reasoning.

By Fedor Kitashov, Jo\~ao Carreira, Shiry Ginosar, Dima Damen, Andrew Zisserman, Viorica P\u{a}tr\u{a}ucean
arXiv Machine Learning
Oct 9

Brain foundation model-guided source-selective domain adaptation for cross-subject EEG decoding

The paper introduces BFM-MSDA, a brain foundation model-guided multi-source domain adaptation framework for cross-subject motor‑imagery EEG decoding. It first uses pretrained brain foundation model representations to estimate source–target compatibility and select relevant source subjects. A relevance-weighted dual alignment strategy then aligns marginal and conditional feature distributions, achieving higher accuracies on two public MI‑EEG benchmarks and demonstrating the benefits of informed source selection.

By Jinzhou Wu, Baoping Tang, Qikang Li, Yi Wang, Cheng Li, Shujian Yu
arXiv Computer Vision
Oct 9

VGGTWorld-VLA: Intent-Conditioned 3D World Evolution for Autonomous Driving

VGGTWorld-VLA extends the VGGT-World model by adding intention-conditioned control to 3D world evolution for autonomous driving. It introduces an action–semantic conditioning mechanism that injects driving semantics and ego‑motion into the future‑token stream, allowing different future geometry predictions for the same observed scene under alternative actions. A geometry–language–action bridge further integrates historical geometry, VLA semantic features, and maneuver/trajectory representations to jointly condition future geometry prediction, achieving competitive performance on NAVSIM and demonstrating the benefits of semantic and action conditioning.

By Zhaoyang Liu, Kun Jiang, Ziying Song, Diange Yang
arXiv Computer Vision
Oct 9

PAM-ToD: Plug-and-Play Appearance Modeling for Cross-Time-of-Day 3D Gaussian Splatting

PAM-ToD is a lightweight plug‑in that adapts a pre‑trained 3D Gaussian Splatting road scene to a new time of day by learning color corrections while keeping the original 3DGS parameters fixed. It scales each Gaussian’s existing color to model illumination changes and adds a brightness term for effects such as street lamps, learning these corrections from a few anchor images without separately estimating albedo and illumination. The method includes a spatial regularization to avoid abrupt changes and is evaluated on the CARLA-ToD benchmark, achieving higher PSNR and lower LPIPS than baselines in both static and dynamic settings.

By Kota Shimomura, Sungho Moon, Tsubasa Hirakawa, Takayoshi Yamashita, Sunghoon Im, Hironobu Fujiyoshi
arXiv Computer Vision
Oct 9

Relative Patch Response Learning for Generalizable AI-Generated Image Detection

The paper introduces Relative Patch Response Learning (PRL) to detect AI‑generated images by comparing patch responses across two mixed views of aligned real‑generated image pairs. PRL uses a relative response objective, a reference coherence objective, and an area ranking objective to address contextual shifts caused by self‑attention, providing precise supervision for patch‑level detection. Experiments show PRL outperforms prior methods by 4.3% and 5.9% in average balanced accuracy on standard and in‑the‑wild benchmarks.

By Tianyu Wang, Ouxiang Li, Yanbin Hao, Zhenhua Tang, Shuo Wang
arXiv Computer Vision
Oct 9

What 30,000 Hours of Ego-centric Video Does Not Teach

The paper investigates how scaling ego‑centric human video data affects world model performance. Using 30,000 hours of video across diverse scenes, the authors find that while agent‑level fidelity improves markedly with more data, object‑level fidelity remains low and improves slowly. They demonstrate that careful visual conditioning and a new supervision scheme can shift model capacity toward object dynamics, yet a substantial gap persists, indicating that data quantity alone is insufficient for full world modeling.

By Jiahua Dong, Anurag Bagchi, Yash Jangir, Muhammad Zubair Irshad, Sergey Zakharov, Martial Hebert, Homanga Bharadhwaj, Yu-Xiong Wang, Vitor Campagnolo Guizilini, Pavel Tokmakov
arXiv Computer Vision
Oct 9

OmniDex: Scaling Dexterous Hand Grasping to Diverse Cluttered Scenes

OmniDex presents a large-scale benchmark for dexterous hand grasping, featuring over 2.6 million cluttered scenes and 0.4 B scene-specific grasp ground truths generated via a scalable seed-and-filter strategy. The paper introduces the OmniDex model, which uses Soft Winner-Takes-All learning, human-inspired physical constraints, and physics-driven ranking to address multimodality and precision errors in generative grasp models. Experimental results demonstrate that OmniDex achieves state‑of‑the‑art performance and strong generalization across diverse scenes, views, and unseen objects.

By Naiyu Fang, Zhongjin Luo, Yuxin Mo, Siyuan Huang, Jianbo Liu, Yufei Liu, Zheyuan Zhou, Chenkai Jin, Xiaogang Wang, Hongsheng Li