Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Computation and Language
Oct 16

Reasoning Error from Known Fact: Step-Level Self-Consistency Group Relative Policy Optimization for LLMs

The paper introduces Step-level Self-Consistency Group Relative Policy Optimization (SSC‑GRPO), a method that assigns rewards to individual steps in a reasoning trace by evaluating self‑consistency across multiple rollouts. It targets Context‑Sensitive Factual Hallucinations—errors that arise when a model has the correct knowledge but produces false statements due to contextual interference. SSC‑GRPO outperforms previous approaches on mathematical reasoning benchmarks and hallucination leaderboards, offering a new way to detect and reduce hallucinations in large language model reasoning.

By Xiaomeng Hu, Jiaqi Hu, Hao Chen, Qi Zhang, Zhanming Shen, Wentao Ye, Junbo Zhao
arXiv Computation and Language
Oct 16

M$^3$R-Bench: A Unified Benchmark for Evidence-Grounded Multimodal Metaphor Understanding

M$^3$R-Bench is a new benchmark comprising 1,000 human‑annotated image‑text pairs that jointly label metaphor occurrence, Target‑Source mapping, and sentiment, and provide step‑by‑step explanations for evidence identification, mapping, and sentiment inference. Existing models often rely on textual shortcuts and produce mappings that conflict with visual evidence, as shown by evaluations on this dataset. To address this, the authors propose M$^3$R‑Reasoner, a model that uses curriculum‑based reasoning supervision and task‑aware reinforcement learning, achieving a 37.90% Joint Label Accuracy and outperforming both proprietary baselines and GPT‑5.5 on evidence and sentiment justification.

By Hong Jiang, Junnan Zhu, Jingwang Huang, Xiao Sun, Yuming Yang, Jiang Zhong, Ruirui Chen, Jingman Shi, Hao Wu, Nayu Liu, Xinyi Jiang, Kaiwen Wei
arXiv Computation and Language
Oct 16

Cost-Efficient RAG for Entity Matching with LLMs: A Blocking-based Batch Processing Framework

The paper introduces CE‑RAG4EM, a cost‑efficient framework that applies retrieval‑augmented generation (RAG) to entity matching (EM) by using blocking‑based batch retrieval and generation. It demonstrates that, in a zero‑shot setting, CE‑RAG4EM achieves comparable or better matching quality than naive RAG while significantly reducing end‑to‑end overhead. Experiments on nine real‑world EM benchmarks show that the approach closes the knowledge grounding gap for LLM‑based EM, especially on datasets with ambiguous or sparse attributes, and provides clear guidance on balancing accuracy and computational cost through blocking size and retrieval granularity.

By Chuangtao Ma, Zeyu Zhang, Arijit Khan, Sebastian Schelter, Paul Groth
arXiv Computation and Language
Oct 16

The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement

The paper proposes a new paradigm for Earth observation (EO) modeling in which a general-purpose coding agent automatically builds and recursively improves models by interacting with a harness and a verifier. Any verifiable EO task can be framed with a harness that defines permissible operations and a verifier that assesses success, enabling the agent to learn from labeled benchmarks, independent sensors, physical models, or repeated observations. On six diverse EO benchmarks—including optical, SAR, and multispectral time-series imagery—the agent-built models outperform state‑of‑the‑art methods on every task, achieving superior land‑cover mapping using only 11% of the training labels.

By Sijie Zhao, Zeqi Zhou, Heli Qi, Xuanchao Fu, Zhuo Zheng, Junshi Xia, Junjue Wang, Weihao Xuan, Naoto Yokoya
arXiv Computer Vision
Oct 16

VisualTrust: Video Reasoning under On-Screen Claim Framing

VisualTrust introduces a counterfactual benchmark to evaluate how multimodal large language models handle on‑screen claims that may conflict with visual evidence. The benchmark contains 5,063 cases across 14 claim‑fact classes, each featuring a question paired with three video versions: no claim, accurate claim, and misleading claim. Results show a drop in accuracy from 90.1% to 51.9% when a misleading claim is present, and a new training‑free agent, DEAR, improves accuracy under misleading claims to 81.8–85.2% while reducing trap‑answer selection.

By Huachao Zhu, Junjielong Xu, Sicheng Li, Liang Yang, Mingtao Chen, Zhichao Hu, Li Jiang, Pinjia He
arXiv Computer Vision
Oct 16

SpaceCache: Intermediate Feature Reuse for Efficient Semantic Segmentation on Point Cloud Streams

SpaceCache is a training‑free feature cache that reuses intermediate features from point‑cloud transformers across consecutive frames, allowing tokens of previously seen regions to skip transformer blocks and only compute new tokens. The method limits cache entry age and evicts low‑confidence entries to maintain reliability, and it operates once per encoder stage at negligible cost. Experiments on outdoor and indoor datasets (nuScenes, Waymo, ScanNet) with Utonia and LitePT‑B backbones show up to 1.65× speedup with less than 1‑point mIoU loss, outperforming token‑merging approaches.

By Dongheon Yang, Beomseok Kang, Jae-Joon Kim
arXiv Computer Vision
Oct 16

FocusOPD: Structure-Guided On-Policy Distillation for Diffusion Models

FocusOPD introduces a structure‑guided on‑policy distillation method that adds token‑wise directional feature alignment to deep Transformer layers during early denoising steps. This approach addresses the mismatch between teacher and student representations that persists after standard OPD distillation. Experiments on rectified‑flow backbones show that FocusOPD improves performance on GenEval, PickScore, and LAION‑Aesthetic benchmarks, and ablations confirm that selective deep and early supervision is more effective than full‑network alignment.

By Xinrui Lin, Shumin Wang, Suqi Song, Fan Li, Yanyong Zhang
arXiv Computer Vision
Oct 16

GeoPano: Towards Geometrically Accurate 3D Reconstruction from a Single Panorama

GeoPano is a feed‑forward framework that reconstructs geometrically accurate 3D point clouds from a single panoramic image. It introduces a panoramic‑aware attention aggregator with a novel Pano‑Attention mechanism to exploit spherical geometry and reduce cross‑view inconsistencies. A hybrid supervision strategy enforces both fine‑grained local detail in perspective views and global structural coherence in spherical space, leading to superior reconstruction accuracy, consistency, and visual fidelity compared to prior state‑of‑the‑art methods.

By Jing OU, Zidong Cao, Liaoyuan Fan, Zhuoxiao Li, Shuai Zhang, Yinrui Ren, Dongli Wu, Haoang Li, Hui Xiong, Wufan Zhao
arXiv Computer Vision
Oct 16

Mixture of Multi-Sequence Feature Difference for Contrast-Enhanced MRI Synthesis

The paper presents a method for synthesizing contrast‑enhanced MRI (CE‑MRI) from non‑contrast sequences using a novel Mixture of Feature Difference (MoFD) module. MoFD dynamically weights and aggregates features from individual sequences and their explicit differences, improving image fidelity and perceptual quality over baseline approaches. Experiments on UCSF‑PDGM and UPenn‑GBM datasets show that the synthesized images reconstruct tumor regions well and yield segmentation performance comparable to real CE‑MRI, with regression‑based training achieving higher structural fidelity.

By Jonghun Kim, Junmo Kwon, Hyunjin Park
arXiv Computer Vision
Oct 16

FuseAdapt: Parameter-Efficient Multimodal Fusion for Semantic Segmentation with Missing Modalities

FuseAdapt is a parameter‑efficient framework for multimodal semantic segmentation that performs fusion within the adaptation space of a frozen vision foundation model (VFM). Instead of a separate fusion network, each modality receives a lightweight adaptation branch, and only low‑rank residual activations are routed across modalities inside the frozen backbone. The method also introduces Predictive Modality Modeling (PMM) to handle missing modalities during training, achieving significant mIoU gains on three benchmarks.

By Xin Zhang, Robby T. Tan
arXiv Computer Vision
Oct 16

LAMP: Layerwise Adaptive Mixing of Predictors for Training-Free Few-Shot Vision-Language Adaptation

LAMP (Layerwise Adaptive Mixing of Predictors) is a training‑free framework that fuses global and regional information from multiple encoder layers for few‑shot vision‑language adaptation. It uses a single forward pass per view to generate descriptors, then applies global and support‑calibrated regional kernel predictors that independently fit label residuals. Validation data select the optimal mixture in class‑score space, preserving native feature dimensions without learned projections, and an extended variant LAMP‑F further adapts text‑anchored class prototypes for improved accuracy across eleven datasets with ViT and ResNet backbones.

By Zhipeng Ye, Feng Jiang, Qiufeng Wang, Hao Li
arXiv Computer Vision
Oct 16

Beyond Pairwise Repulsion: Collective Diversity via Determinant Maximization in Diffusion Sampling

The paper introduces DetNO, a method that enhances diversity in text-to-image diffusion models by maximizing the log-determinant of an RBF Gram matrix over initial noise vectors. DetNO treats diversity as a collective property of the entire batch, using a volume maximization objective that self-attenuates as diversity improves, and includes an anchor-based fidelity term to keep samples realistic. Experiments on SD1.5, SDXL, and SD3 backbones show that DetNO outperforms existing pairwise repulsion methods in intra-batch diversity metrics and yields statistically significant accuracy gains in few-shot classification tasks when used for synthetic data augmentation.

By Kwanyoung Lee, SeungJu Cha, Yebin Ahn, Sungho Koh, Dong-Jin Kim
arXiv Computer Vision
Oct 16

Grounded Human-Attributed Description and Activity Recognition in Videos

The paper introduces Grounded Human-Attributed Description and Activity Recognition (GHADAR), a task that requires describing the characteristics and activities of every person in a video given their locations. It presents a new dataset derived from AVA-Actions with open‑set captions, a method that constrains cross‑attention masks in vision‑language models to better use grounding information, and a holistic VLM‑based evaluation schema that compares concepts directly between video and predictions. Experiments show the proposed method surpasses state‑of‑the‑art VLMs and highlight limitations of existing evaluation metrics.

By Jay N. Paranjape, Jurijs Nazarovs, Rohan Sarkar, Yue Guo, Lipeng Ke, Sankar Venkataraman, Vishal M. Patel, Nataraj Jammalamadaka
arXiv Computer Vision
Oct 16

Not Just a Subject: Capturing Adaptive Scene-Specific Variance for Visual Storytelling

EOS-Story is a training‑free framework that improves visual storytelling by balancing identity consistency with scene‑specific adaptation. It separates narrative prompts into identity and scene sub‑prompts and introduces Cross‑Contextual Bridging to inject appearance cues into the scene embedding. In the vision space, Adaptive Identity Preservation aligns identity features to the scene scale and uses Identity‑Aware Self‑Attention to avoid structural artifacts, achieving competitive results on the extended Consistory+ benchmark.

By SeungJu Cha, Ye-Chan Kim, Kwanyoung Lee, Dong-Jin Kim
arXiv Computer Vision
Oct 16

Does Peer Observation Help? Vision-Sharing Collaboration for Vision-Language Navigation

The paper introduces Co‑VLN, a minimalist, model‑agnostic framework that lets agents in Vision‑Language Navigation share structured perceptual memory when they independently visit the same locations. By exchanging observations, each agent expands its receptive field without extra exploration. Experiments on the R2R benchmark with DUET and MapGPT show that vision‑sharing significantly improves performance, suggesting a promising direction for collaborative embodied navigation.

By Qunchao Jin, Yiliao Song, Qi Wu
arXiv Computer Vision
Oct 16

PhyMix: Towards Physically Consistent Single-Image 3D Indoor Scene Generation with Implicit--Explicit Optimization

PhyMix introduces a Physics Evaluator that checks geometric priors, contact, stability, and deployability through nine sub-constraints, establishing a benchmark for physical consistency in single-image 3D indoor scene generation. The framework uses implicit alignment via Scene-GRPO and explicit refinement through verifier-guided agentic repair to incorporate evaluator feedback during training and inference, improving physical plausibility while preserving scene content. Experiments demonstrate enhanced reconstruction quality and physical plausibility, with qualitative results on stylized and real-world images showing the method’s broader applicability.

By Dongli Wu, Jingyu Hu, Ka-Hei Hui, Xiaobao Wei, Chengwen Luo, Jianqiang Li, Zhengzhe Liu
arXiv Computer Vision
Oct 16

WaveLLM: Wavelet-Domain LLM Entropy Modeling for Lossless Image Compression

WaveLLM introduces a wavelet-domain entropy model that leverages large language models for lossless image compression. By decomposing images into multi-level wavelet subbands, it transforms heterogeneous prediction tasks into structured, statistically homogeneous subproblems, enabling more effective autoregressive probability estimation. Using a dedicated numerical interface and task-adaptive prompting, WaveLLM, built on Llama‑3.2‑3B with LoRA, achieves state‑of‑the‑art bpsp results on Kodak, CLIC.pro, and DIV2K, outperforming both PixelCNN‑based codecs and recent LLM‑based compressors.

By Yue Yu, Bowei Kang, Changsheng Gao, Dong Liu, Xiaoyan Sun
arXiv AI
Oct 16

Causal Integrated Gradients: Scalable Feature Attribution Along Causal Paths

The paper introduces Causal Integrated Gradients (CIG) and Causal Expected Gradients (CEG), which extend integrated gradients by incorporating known causal relationships among input features. By routing the integration path along a causal graph, the methods decompose each feature’s attribution into direct and mediated effects, enabling scalable feature attribution that aligns with true causal influence. Experiments on synthetic data, simulated single‑cell gene expression, and real interferon‑β stimulation data demonstrate that CIG outperforms existing gradient‑based methods in identifying upstream drivers while maintaining computational efficiency.

By Joran Michiels, Anis Ismail, Maarten De Vos, Alejandro Sifrim
arXiv AI
Oct 16

Memorilla: Latent Semantic Memory for LLMs

Memorilla is a memory system that encodes an entire document collection into reusable embeddings and retrieves query‑specific information as a fixed set of memory tokens for a frozen large language model. Using only 16 memory tokens, it outperforms baselines such as Mem0 and MemoRAG on most benchmarks, achieving higher accuracy on FactKG while drastically reducing token usage. In agentic settings, Memorilla helps prevent wandering and boosts success rates in a TextWorld environment, demonstrating its utility for continual learning and long‑context interactions.

By Adibvafa Fallahpour, Parsa Idehpour, Vignesh Kothapalli, Nikita Mounier, Shirley Wu, Shayan Pardis, Jure Leskovec
arXiv AI
Oct 16

LOADBench: A Large Open Analog Dataset and Benchmark for Machine Learning in IC Design

The paper introduces LOADBench, a large open dataset and benchmark suite for machine learning in analog integrated circuit design. It contains over 100 operational amplifier topologies from the SkyWater130 PDK, each with detailed annotations, sizing constraints, testbenches, and thousands of Pareto-optimal sizing-performance pairs generated by NSGA-II. Six standardized benchmark tasks—inverse sizing, topology selection, sub‑circuit classification, structural circuit generation, Pareto‑membership prediction, and multi‑objective optimization—are defined with precise protocols, and all data, annotations, and reference implementations are released under an open license.

By M. Leibl, F. Azevedo, S. Uhlich, A. Bonetti, A. Venkitaraman, C. Hsieh, P. Pham, L. Servadei, H. Graeb, R. Martins