Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Machine Learning
Oct 5

Differentiable Koopman Operator for Contrastive Learning on Dynamic Graphs

The paper introduces KAIROS, a self‑supervised framework that embeds a differentiable Koopman operator into a dynamic graph contrastive learning loop. By linearizing temporal evolution in the embedding space, KAIROS jointly optimizes raw node features and a graph‑diffused structural view using multi‑granularity contrastive objectives across time windows. For anomaly detection, it combines Koopman prediction residuals with temporal inconsistency and local neighborhood deviation, achieving state‑of‑the‑art results on nine dynamic graph benchmarks and competitive performance on unsupervised node classification.

By Md Abrar Jahin, Taufikur Rahman Fuad, Md Rizwan Parvez
arXiv Computer Vision
Oct 5

Found but Not Read: When Extracted Text Closes the Retrieval-Reading Gap in Document Vision-Language Models

The paper investigates the gap between retrieval and reading in document vision‑language models, showing that even when the correct page is retrieved, the model often fails to use the text on that page. By comparing answers derived from page images alone versus images plus extracted OCR text, the authors demonstrate that adding OCR text can improve strict accuracy by 13 to 16 points on their FoveDoc‑Bench benchmark. The study also reveals that OCR benefits textual evidence but not charts or figures, and that its advantage diminishes as retrieval quality worsens.

By Qingtao Xia, Siyao Cheng, Jiahua Bao, Jiaxing Du, Jie Liu
arXiv Computer Vision
Oct 5

Bridging Research and Practice: A Systematic Evaluation of Generalist and Dermatology-Specific Models in Clinical Skin Lesion Classification

The paper evaluates how well generalist and dermatology-specific machine learning models perform on diverse skin lesion datasets, including dermoscopic images and smartphone photographs. It benchmarks a range of architectures—general-purpose vision-language models, foundation models, and task-specific dermatology classifiers—under conditions of distribution shift, modality change, and demographic variability. The study quantifies the performance gap between current state‑of‑the‑art models and the robustness needed for safe, equitable clinical deployment.

By Emanoel dos Santos, Kelvin Cunha, Rodrigo Mota, Fabio Papais, Thales Bezerra, Natalia Lopes, Erico Medeiros, Shirley Cruz, Jessica Araujo, Paulo Borba, Tsang Ing Ren
arXiv Computer Vision
Oct 5

Interpretable Deepfake Detection in Videos via Explicit Forensic Features and Temporal Modeling

This paper presents an interpretable deepfake detection framework that explicitly encodes physically grounded forensic cues to analyze spatially and temporally coherent facial features in video sequences. The method transforms videos into identity-consistent facial trajectories, segments them into fixed-length temporal windows, and represents each frame with 68 structured descriptors across photometric, textural, geometric, and compression domains. These descriptors are processed by an LSTM to capture temporal dependencies, achieving strong F1-scores on four benchmark datasets and demonstrating robust cross-dataset generalization.

By Chahira Benhama, Mohand Sa\"id Allili, Assia Hamadene
arXiv Computer Vision
Oct 5

MoSE3: Learning World-Space SE(3) at Every Pixel

MoSE3 is a feed‑forward model that predicts dense SE(3) motion—full 6‑DoF rigid transforms—at every pixel from monocular RGB video, providing rotation, translation, and grouping information simultaneously. It achieves this by jointly learning 3D point tracks and rigidity embeddings, then differentiably fitting transforms within soft rigid clusters. The authors also release Art‑Kubric, a large synthetic dataset with dense SE(3) and rigidity labels, and demonstrate state‑of‑the‑art performance on both rigid and articulated benchmarks, with strong generalization to real‑world videos.

By Jiahuan Cheng, Zhiyi Li, Tian Xia, Ruojin Cai, Yilun Du, Qianqian Wang
arXiv AI
Oct 5

When Terminal-Agent Training Stalls: Demystifying Data Generation and Verification Challenge

The paper investigates why training terminal agents often stalls when using a meta‑agent like Claude Opus to generate tasks and verifiers. It identifies three failure modes—benchmark invalidity, harness brittleness, and reward misalignment—and shows that prompt redesign and context extension can improve solvability by 5.6×, yet a 9B model still tops out at 81.3% mean pass@2. Adding hard tasks drops performance to 20.6%, highlighting that solvability depends on the model used.

By Xi Qin, Isabel Kurth, Xin Cui, Elin Park, Alexander Schaefer, Yaad Oren
arXiv AI
Oct 5

HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents

HyperBrowseComp is a multilingual and multimodal web‑browsing benchmark featuring 423 manually authored, human‑validated questions in 13 languages. The questions are intentionally difficult, requiring users to locate obscure evidence, follow multi‑step clue chains, or inspect heterogeneous sources such as videos, scanned documents, images, or maps. The benchmark filters out easier questions by testing models without internet access and evaluates performance using provider‑native search and a shared external retrieval harness, with a human evaluation on a sample to contextualize model effort.

By Alham Fikri Aji, Faiz Rizki Ramadhan, Zayd M. K. Zuhri, Seung Hun Eddie Han, Ryandito Diandaru, Qinrong Cui, Jan Christian Blaise Cruz, Badrinath Chandana, Peerawat Chomphooyod, Ahmed Attia, Jonibek Mansurov, Emilio Villa-Cueva, Canh Duong Nguyen, Imran Turganov, Minghao Wu, Peerat Limkonchotiwat, Irina Nikishina
arXiv Computation and Language
Oct 5

Understanding Clinical Cognitive Dialogues Using Large Language Models

The paper introduces a de‑identified corpus of 33 in‑person cognitive assessment conversations, comprising 8,250 utterances annotated for three speaker roles and 56 dialogue acts. The authors benchmark large language models on fine‑grained dialogue‑act classification and next‑patient‑utterance generation, finding that instruction tuning and reasoning‑aware fine‑tuning improve performance but that models still struggle with closely related dialogue acts. The corpus and benchmark are presented as tools to measure interaction structure in cognitive assessments and to support future research on conversational markers, clinician education, and validated simulated patients.

By Vishalakshi Arumugam, Dan Schumacher, Veronica Rammouz, Erfan Nourbakhsh, Enrique Gonzalez Guerrero, Jeremy Davis, Anthony Rios
arXiv Machine Learning
Oct 5

Test-time Multi-agent Coordination by Decomposed Value Gradient Flow

The paper introduces SCOUT, an offline multi-agent reinforcement learning framework that combines a generative behavioral prior with a decomposed value function for test-time action refinement. SCOUT uses optimal unified transport and Stein variational gradient descent to steer behavioral samples toward high-value regions, with the number of transport steps providing adaptive scaling. The authors prove a single-term KL bound on the joint soft-value gap under the individual-global-max principle and demonstrate that SCOUT outperforms existing methods on both discrete and continuous offline MARL benchmarks, including offline-to-online settings.

By Dongsu Lee, Haoran Xu, Amy Zhang
arXiv Computer Vision
Oct 5

EVEWorld: Physical Evolution Supervision for Embodied World Models

EVEWorld introduces a physical evolution-supervision framework for embodied world models, addressing the issue of Model Laziness by focusing on physical consistency rather than visual fidelity. The framework comprises Instance-Guided Restoration (IGR) to enforce instance consistency and Temporal Instance Alignment (TIA) to align target instances across adjacent frames. Experiments on DreamGenBench, EWMBench, and PBench show an 87.5% reduction in the Model Laziness Rate (MLR) compared to GigaWorld-0, and the model ranks 6th in JEPA Similarity on the WorldArena 2.0 Track 1 leaderboard.

By Kaiqi Wang, Songxin Zhang, Zejian Xie, Xiao Xiong, Zhuoyang Song, Ziwei Wu, Jun Yu Lu, Yitan Teng, Ziying Song, Jiaxing Zhang
arXiv AI
Oct 5

NARA: Anchor-Conditioned Representation Learning for Heterogeneous Vector Geoentities

The paper introduces NARA, a self‑supervised framework for learning representations of heterogeneous vector geospatial entities such as roads, buildings, and points of interest. NARA uses spatial‑context‑aware attention to model spatial autocorrelation, masked semantic modeling, geometry‑aware relation modeling, and relation‑conditioned regularization to capture diverse spatial relations. The resulting frozen, task‑agnostic encoder outperforms specialized state‑of‑the‑art methods on traffic‑speed prediction, building‑function classification, and next point‑of‑interest prediction.

By Jina Kim, Gengchen Mai, Lingyi Zhao, Khurram Shafique, Yao-Yi Chiang
arXiv AI
Oct 5

Reach or Solve? Deep Diving into Agentic RL Gains with Checkpoint Handoffs

The paper introduces checkpoint handoff, an evaluation protocol that separates an agent’s ability to reach useful states from its ability to complete tasks in reinforcement learning. By using one checkpoint as a reacher up to a handoff point and another as a solver from the same replayed history, the authors can measure Reach (how often states within a fixed number of actions from success are achieved) and Solve (how often the task is completed from those states). Experiments on TravelPlanner and ALFWorld show that switching the solver from supervised fine‑tuning to RL yields larger gains when RL is used as the reacher, indicating that RL more effectively finds solvable states.

By Xuan Liu, Jingbin Qian
arXiv AI
Oct 5

Robust Adversarial Quantification via Conflict-Aware Evidential Deep Learning

The paper introduces Conflict‑Aware Evidential Deep Learning (C‑EDL), a lightweight post‑hoc method that improves uncertainty quantification for deep learning models. C‑EDL applies diverse, task‑preserving transformations to each input and uses representational disagreement to adjust predictions, thereby reducing overconfident errors on adversarial and out‑of‑distribution data. Experiments demonstrate that C‑EDL outperforms existing Evidential Deep Learning variants and baselines, achieving up to 55 % reduction in coverage for OOD data and 90 % for adversarial data across multiple datasets and attack types.

By Charmaine Barker, Daniel Bethell, Simos Gerasimou
arXiv Computer Vision
Oct 5

DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

DEPICT is a new training‑free metric for evaluating text‑to‑image alignment. It replaces fixed reference answers with an agreement rule that compares image‑based and caption‑only responses, weighting questions by how decisively the caption determines them. By merging this agreement score with a holistic score, DEPICT improves negation accuracy dramatically and outperforms existing training‑free metrics while matching or exceeding fine‑tuned evaluators on several benchmarks.

By Vasco Ramos, Sandra Godinho Silva, Joao Magalhaes, Ricardo Rei, Pedro Henrique Martins
arXiv Computer Vision
Oct 5

FRUC: Feedforward Dynamic Scene Reconstruction from Uncalibrated Collaborative Driving Views

FRUC is a feedforward 3D Gaussian Splatting framework that reconstructs dynamic scenes from uncalibrated collaborative driving views. It uses a visual‑grounded geometric Transformer backbone for one‑shot, calibration‑free inference and introduces an ego‑centric causal occlusion field to model occlusion evolution across agents. The method performs cross‑agent integration as a deterministic residual denoising process, achieving state‑of‑the‑art rendering quality and efficiency on V2X‑Real and UrbanIng‑V2X datasets.

By Yihang Tao, Yu Guo, Zhengru Fang, Haonan An, Yuguang Fang
arXiv Computation and Language
Oct 5

Automatic Evaluation of Mental Health Stigma in Online Communication

The paper presents a new benchmark for automatically evaluating mental health stigma in online text, featuring a fine‑grained taxonomy that covers stigma mode, domain, and specific components across multiple mental health conditions. The authors annotate naturally occurring news and social media posts and test large language models and classifiers for sentiment, toxicity, and hate speech, finding that these models poorly capture stigma and often overpredict it without explicit rules. The benchmark, annotations, exemplar cases, and code are publicly released on GitHub.

By Naomi Baes, Jemima Kang, Nick Haslam, Chris Groot, Alsa Wu, Luc Raszewski, Yulia Otmakhova
arXiv Computation and Language
Oct 5

Adaptive Mutual Distillation for Balanced Multi-Task Post-Training of Large Language Models

The paper introduces Adaptive Mutual Distillation (AMD), a post‑training framework that jointly trains two large language models using different task‑balancing strategies. AMD evaluates and selects distillation weight adjustments via short training probes and task‑wise validation, leading to models that outperform supervised fine‑tuning baselines on six benchmarks across three backbones. Merging the two AMD models further improves performance, surpassing multi‑task fine‑tuning by an average of 2.91 points.

By Baohang Li, Xiaocheng Feng, Yichong Huang, Chengpeng Fu, Wenshuai Huo, Zekun Zhou, Zekun Yuan, Tingjia Zhang, Bing Qin
arXiv Computation and Language
Oct 5

OmniConfess: Eliciting Token Confessions to Mitigate Omni-Modal Hallucination

OmniConfess is a training‑free method designed to reduce hallucinations in omni‑modal large language models (OmniLLMs) that handle text, images, audio, and video. The approach fixes a candidate response and re‑scores it at token resolution while selectively intervening on evidence from each modality, producing a token‑by‑channel confession that shows which evidence supports each part of the response. Using this confession, OmniConfess preserves grounded content and corrects commitments that rely on irrelevant or contradictory evidence. The authors evaluated the method on OmniHalluBench, a 3,540‑example benchmark drawn from six datasets across multiple modalities and tasks, and found that OmniConfess mitigates hallucinations across diverse settings.

By Huiqiang Rong, Haoran Luo, Hui Feng, Zhonghong Ou, Kaiwen Xue, Guoxin Zhang, Yifan Zhu
arXiv Machine Learning
Oct 5

MuLoRA: Spectrally Balanced Low-Rank Adaptation for Continual Learning

MuLoRA introduces a spectrally balanced low‑rank adaptation method for continual learning, addressing the issue of spectral plasticity collapse where adaptation energy concentrates in few singular modes. It uses historical whitening to identify task‑adaptive input directions and approximate polar orthogonalization of momentum updates to reduce spectral concentration, ensuring effective utilization of the low‑rank space. Experiments on five class‑incremental benchmarks and eight incremental settings show MuLoRA achieving the highest mean accuracy in 15 of 16 metrics.

By Junkang Liu
arXiv Machine Learning
Oct 5

Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures

The paper introduces a stability theory for Joint-Embedding Predictive Architectures (JEPAs), showing that training dynamics involve a driving force and a decay effect that determine representation collapse. By linearising the gradient flow, the authors derive a per‑mode stability ratio that separates data‑side and predictor‑side contributions, predicting a phase boundary confirmed across 800 configurations. Using this insight, they propose ResidualPred, a transformer predictor that biases attention toward the identity at initialization, improving representation rank and downstream accuracy on tabular and image benchmarks.

By Jos\'e Lucas De Melo Costa, Seong Woo Ahn, Fabrice Popineau, Arpad Rimmel, Bich-Li\^en Doan