Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv Computation and Language
Oct 7

Enhancing High-order Interaction Awareness in LLM-based Recommender Model

The paper introduces ELMRec, an enhanced LLM-based recommender that improves high-order interaction awareness by using whole-word embeddings to better interpret graph-constructed user-item interactions without graph pre-training. It also identifies that LLMs tend to prioritize earlier user interactions over recent ones and proposes a reranking solution. ELMRec achieves superior performance compared to state-of-the-art methods in both direct and sequential recommendation tasks.

By Xinfeng Wang, Jin Cui, Fumiyo Fukumoto, Yoshimi Suzuki
arXiv Computer Vision
Oct 7

2D Spatial Reasoning with Adaptive Neural Cellular Automata

The paper introduces Adaptive Neural Cellular Automata (aNCA), a new architecture that employs deformable convolutions to dynamically adjust its receptive field and iteratively reason over 2D spatial relationships in grid-like data such as images. aNCA is designed to address the limitations of current learning approaches in spatial reasoning tasks. Experiments on public benchmarks demonstrate state‑of‑the‑art performance and strong generalization on image‑based puzzles like Sudoku and maze shortest‑path problems.

By Martin Spitznagel, Janis Keuper
arXiv Computer Vision
Oct 7

ALIVE: Interaction-Aligned Object Insertion for First-Frame-Guided Video Editing

ALIVE is a new framework for first‑frame‑guided video editing that enables inserted objects to participate in coherent interactions with the source video, such as being picked up or manipulated. The authors curated 35,800 editing pairs from 3D‑rendered, model‑generated, and real‑world videos, and trained a vision‑language model to predict interaction guidance from the edited first frame and an object name. ALIVE outperforms the strongest baseline by 43.9% on overall performance and 4.4% on a general video object insertion benchmark, with VLM‑predicted guidance further improving interaction fidelity.

By Zhenghong Zhou, Zhe Lin, Jiebo Luo, Yuqian Zhou
arXiv Computer Vision
Oct 7

WareFly-VLA: A Vision-Language-Action Framework for UAV Navigation and Human Tracking in Smart Warehouses

WareFly-VLA is a photorealistic vision‑language‑action framework and dataset for UAV navigation and human tracking in smart warehouses. It comprises 507 human‑teleoperated flight episodes with 8,504 high‑resolution RGB transitions, each paired with a natural‑language description of a target worker and a synchronized four‑degree‑of‑freedom control command. The benchmark evaluates four open‑source VLA architectures on tasks such as target approach and person following, revealing significant challenges for language‑conditioned aerial control under realistic warehouse conditions.

By Thinh D. Le, Son T. Nguyen, Duong Q. Nguyen, Dung D. Le, Ngo Anh Vien, H. Nguyen-Xuan
arXiv Computer Vision
Oct 7

Beyond the Embedding Bottleneck: Adaptive Retrieval-Augmented 3D CT Report Generation

The paper identifies a representational bottleneck in 3D CT embeddings that limits pathology coverage in automated radiology reports. It introduces AdaRAG-CT, an adaptive retrieval-augmented framework that supplements visual features with controlled textual retrieval to improve report generation. On the CT-RATE benchmark, AdaRAG-CT achieves state‑of‑the‑art clinical efficacy, raising Clinical F1 from 0.420 to 0.480.

By Renjie Liang, Yiling Ma, Yang Xing, Zhengkang Fan, Jinqian Pan, Chengkun Sun, Li Li, Kuang Gong, Jie Xu
arXiv Computer Vision
Oct 7

PlotPick: AI-powered batch extraction of numerical data from scientific figures

PlotPick is an open‑source tool that employs vision‑language models (VLMs) to batch‑extract tabular data from scientific figures. In benchmarks on synthetic chart datasets (ChartX and PlotQA), nine VLMs generally outperform the dedicated chart‑to‑table model DePlot, especially on box plots and horizontal bar charts. The tool’s figure detection, output formatting, and default model (Claude Haiku 4.5) were not evaluated, and accuracy on real biomedical figures remains unverified.

By Tommy Carstensen
arXiv Computer Vision
Oct 7

SalArt-VQA: Diagnosing Whether VLMs Understand Salient Artifacts in Generated Images

SalArt-VQA is a diagnostic benchmark that tests whether vision‑language models can understand salient artifacts in AI‑generated images. It includes 950 images and 3,681 multiple‑choice questions that assess artifact presence, semantic localization, spatial grounding, and evidence‑grounded defect identification. The benchmark reveals that high image‑level detection accuracy can mask failures in grounded understanding, showing a trade‑off between sensitivity and calibration.

By Ruotian Zhang, Xiaoxiao Sun, Junzhe Huang, James Burgess, Serena Yeung-Levy
arXiv Computer Vision
Oct 7

Dataset Biases and Shortcut Learning in Motion-Based AI-Generated Video Detection

The paper examines four leading motion‑based AI‑generated video detectors and finds that three of them suffer from preprocessing and sampling biases that inflate their reported performance. These detectors rely heavily on motion patterns—specifically, the lower inter‑frame movement typical of synthetic videos—so their accuracy drops to near random when tested on datasets lacking this bias or after simple spatial augmentations. In contrast, a frequency‑based detector remains robust across all datasets, indicating that frequency‑domain methods may generalize better for detecting AI‑generated videos.

By Joren Michels, Lode Jorissen, Nick Michiels
arXiv Computer Vision
Oct 7

RPA: Residual Patch-Token Adapter for Image Retrieval from EEG and MEG

The paper introduces the Residual Patch Adapter (RPA), a lightweight module that aligns EEG and MEG signals with all patch tokens from an intermediate layer of a Vision Transformer, rather than a single global embedding. Ablation studies show that retaining the full set of patch tokens is crucial for effective alignment, while the CLS token adds little value. Using RPA, the authors achieve state‑of‑the‑art Top‑1 accuracies on THINGS‑EEG2 and THINGS‑MEG datasets, and demonstrate that the approach generalizes to other brain encoders and can be swapped with various neural network backbones.

By Yuhui Jin, Yonghao Song, Bingchuan Liu
arXiv Computer Vision
Oct 7

VolS-GS: Relightable Gaussian Splatting with Volumetric Subsurface Scattering

VolS-GS is a relightable Gaussian splatting framework that reconstructs objects from one-light-at-a-time captures and renders them under novel lighting and viewpoints. It addresses the difficulty of modeling non‑local effects such as subsurface scattering by using the spatial support of the Gaussian scene as the domain of a differentiable finite‑volume transport solver, allowing light to propagate through the object's interior. A small network predicts scattering and absorption coefficients for each Gaussian, and the solver redistributes incident light, while a shadow term and regularizer prevent learned shadow and specular terms from dominating the appearance. "whyItMatters":"The approach improves relighting quality on held‑out lights and views across three OLAT benchmarks, demonstrating its effectiveness for realistic rendering of subsurface scattering effects."

By Junyeong Ahn, Jaegul Choo
arXiv AI
Oct 7

Beyond Scalar IoU: Structured Verification from Rollout Groups for Video Temporal Grounding

The paper introduces SUTURE, a structured verification method for video temporal grounding that leverages the joint structure of rollout groups rather than scoring each rollout independently. SUTURE conditions verification on the entire rollout group, using disagreement across rollouts to reweight targets and coverage at each position to redistribute reward mass. Experiments on five benchmarks show that SUTURE consistently improves grounding performance across all IoU thresholds and reduces video-start anchoring in reasoning traces, indicating that the joint structure of a rollout group can provide a more informative temporal verifier.

By Youngjae Cho, Won Young Jhoo, Jongsuk Kim
arXiv AI
Oct 7

Matching Object or Relation? Tracing Abstract Reasoning Inside VLMs

The paper investigates why Vision Language Models (VLMs) struggle with abstract reasoning tasks. Using the Relational Match-to-Sample paradigm and mechanistic analysis, the authors identify four factors—capability tier, model scale, number of objects per scene, and lack of per-object noise—that influence VLM performance. They uncover two competing neural circuits: an early surface-feature circuit and a late relational circuit, and show that ablating relational heads degrades performance more than random heads, suggesting the relational circuit is broadly used.

By Gouki Minegishi, Hiroki Furuta, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
arXiv AI
Oct 7

BluffJAX: Adversarial Imperfect Information Games in JAX

BluffJAX is an open‑source suite of adversarial imperfect‑information games implemented in JAX, designed for high‑throughput simulation and GPU parallelization. It includes classic benchmarks like Texas Hold’Em and Kuhn Poker, as well as newer games such as Bluff, Stud Poker, and Kemps, aiming to broaden research in game‑theoretic reinforcement learning. The authors benchmarked throughput, memory usage, and algorithmic performance across single and multi‑GPU setups, showing scalability to hundreds of millions of samples per second and providing baseline results for future work.

By Aryaman Reddi, Jan Peters, Carlo D'Eramo
arXiv AI
Oct 7

Learning from Revision Consequences: Hindsight Meta-Experience Distillation for Self-Improving Agents

The paper introduces Hindsight Meta-Experience Distillation (HMED), a method that lets self‑improving agents evaluate how revisions to their Meta‑Skills affect skill discovery by re‑executing both the original and revised Meta‑Skills from the same initial state. By comparing these outcomes, HMED creates structured Meta‑Experiences that provide learning signals even for revisions that are not ultimately kept. Experiments on three interactive benchmarks show that HMED consistently outperforms strong baselines, improving skill discovery by focusing on the consequences of the improvement process rather than just branch outcomes.

By Qianhan Feng, Zhongzhen Huang, Yakun Zhu, Xiaofan Zhang, Qi Dou
arXiv AI
Oct 7

How Well Do LLMs Reason with Noisy Evidence? An Active Visual Reasoning Benchmark

The paper introduces VisualNoiseQA, a benchmark that tests large language models (LLMs) on active reasoning tasks involving noisy visual feedback. In this setup, a text-only LLM must solve visual question answering (VQA) problems by repeatedly querying an off-the-shelf vision-language model (VLM) that acts as a stochastic sensor. Each query generates multiple samples, and the resulting empirical uncertainty is exposed via self‑consistency, allowing the LLM to decide what to ask next and when to stop. The benchmark is automatically constructed from diverse VQA sources and two noisy VLMs, retaining only questions where the sensor is inconsistent yet solvable by humans, and evaluates several LLM reasoners on 1,000 instances covering perception, chart understanding, and knowledge-intensive reasoning.

By Bach Nguyen, Zhaonan Li, Mau Son Nguyen, Sanika Chavan, Nilay Kumar, Hong Anh Nguyen, Khoa Vo, Ben Zhou
arXiv AI
Oct 7

SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning

SpeedrunBench is a new benchmark that tests large language model agents on their ability to develop and refine strategies for completing nine different video games as quickly as possible. The benchmark requires agents to repeatedly improve, reflect, and reason over long action horizons to beat both themselves and other players. Experiments show that while agents can approach human records on simple platformers, they lag behind humans on longer, more complex games within realistic resource limits.

By Yoshinari Fujinuma, Keisuke Kamahori, Ryuto Koike, Abdelrahman Madkour, Varun Prashant Gangal, Monty Bichouna, Martyna Markiewicz, Shivani Jain, Duncan Curtis, Rebecca Qian, Anand Kannappan
arXiv AI
Oct 7

ThinkFuse: Trajectory-Aware Test-Time Fusion for Small Reasoning Models

ThinkFuse is a training‑free test‑time fusion framework designed for small reasoning models (SRMs). It selectively intervenes in unreliable reasoning segments by comparing segment‑level uncertainty shifts with trajectory‑level uncertainty trends, and fuses auxiliary reasoning paths into the primary model’s trajectory. Experiments show ThinkFuse outperforms baselines on mathematical and knowledge‑intensive benchmarks, requires fewer fusion triggers, and generates fewer tokens, demonstrating efficient selective triggering.

By Myunghoon Kang, Jungseob Lee, Jaehyung Seo, Heuiseok Lim
arXiv AI
Oct 7

RA-MoWE: Workflow-Affinity Embeddings for Query Clustering and Agentic Workflow Generation

RA-MoWE introduces workflow‑affinity embeddings to cluster queries and guide the creation of reusable expert workflows for large language models. Each embedding captures how well a set of reference workflows solves a query, revealing common reasoning strategies. The framework uses cluster embeddings to initialize and refine specialized workflows, and an encoder predicts embeddings from query text, enabling efficient expert selection without executing reference workflows.

By Qi Cheng, Shengyu Chen, Wei Cheng, Yiqun Xie, Haoyu Wang, Haifeng Chen, Xiaowei Jia
arXiv AI
Oct 7

GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets

GameGo is a framework that converts short game ideas into detailed Product Requirements Documents using industry practices, enabling coding agents to generate complete games from sparse user queries. It employs dynamic compression to keep essential gameplay constraints while allowing design flexibility. The authors built GameGoData with over 55,000 development trajectories and GameGoBench with 124 game queries, training GameGoCoder to outperform baselines and match leading models on gamedev benchmarks.

By Haoyue Yang, Jingyao Li, Zhengfan Wu, Jing Liu, Xuanle Zhao, Kang Liu
arXiv AI
Oct 7

When to Rethink: Learning Multi-Perspective Self-Verification for Vision-Language Models

The paper introduces MOTIVE, a multi-view self-verification framework for vision-language models that evaluates candidate answers from complementary perspectives and learns a reliability score aligned with correctness. By using this score to decide whether to accept an answer or trigger history-guided rethinking, MOTIVE improves reliability and efficiency compared to existing self-verification and self-correction methods. Extensive experiments on diverse multimodal benchmarks and VLM backbones show consistent performance gains and reduced unnecessary reasoning turns.

By Ziquan Zhu, Hanruo Zhu, Si-Yuan Lu, Morris Yu-Chao Huang, Yicheng Lin, Wei Han, Tianlong Chen, Mingyuan Wu, Hanchao Yu, Gaojie Jin, Lu Liu, Bo Sun, Tianjin Huang