Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Computer Vision
Oct 9

DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training

DreamTrue is a multi‑view, cross‑embodiment robot world model that predicts physically plausible videos while faithfully following actions. It overcomes calibration and limited interaction coverage issues by rendering action trajectories into image‑space conditions, applying offline geometric calibration, and using counterfactual post‑training to generate a broader range of future videos. A human‑annotated defect dataset trains a video reward model that guides reinforcement‑learning post‑training, resulting in state‑of‑the‑art action following on AgiBot and a significant drop in interaction defect rates.

By Junyan Li, Ruizhi Li, Yu Liu, Xiangshuo Liu, Mingchao Sun, Hongyu Pan, Mu Xu, Lue Fan, Zhaoxiang Zhang
arXiv Computer Vision
Oct 9

DDL: A Large-Scale Dataset for Deepfake Detection and Localization in Diversified Real-World Scenarios

The paper introduces DDL, a large-scale dataset for deepfake detection and localization, comprising over 1.4 million forged samples generated by 80 distinct deepfake methods. It features comprehensive coverage of generation architectures, varied manipulation modes, diverse scenarios and modalities, and provides fine-grained spatial masks and temporal segments for each forgery. These enhancements aim to offer a more challenging benchmark and support the development of next-generation deepfake detection, localization, and interpretability methods.

By Changtao Miao, Yi Zhang, Weize Gao, Zhiya Tan, Weiwei Feng, Man Luo, Jianshu Li, Ajian Liu, Yunfeng Diao, Qi Chu, Tao Gong, Zhe Li, Weibin Yao, Joey Tianyi Zhou
arXiv Computer Vision
Oct 9

BiomedXPro: Prompt Optimization for Explainable Diagnosis with Biomedical Vision Language Models

BiomedXPro is an evolutionary framework that uses a large language model to automatically generate a diverse set of interpretable, natural‑language prompt pairs for disease diagnosis with biomedical vision‑language models. It addresses the lack of transparency in existing prompt optimization methods by producing prompts that are both understandable and aligned with clinically significant features. Experiments on multiple biomedical benchmarks show that BiomedXPro outperforms state‑of‑the‑art prompt‑tuning techniques, especially in few‑shot, data‑scarce scenarios.

By Kaushitha Silva, Mansitha Eashwara, Sanduni Ubayasiri, Ruwan Tennakoon, Damayanthi Herath
arXiv Computer Vision
Oct 9

Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts

The paper argues that multimodal large language models can answer vision benchmarks using only linguistic cues, revealing non-visual shortcuts. It proposes a Test‑set Stress‑Test (TsT) that trains a text‑only model on the test set to expose exploitable patterns, and introduces Iterative Bias Pruning (IBP) to remove high‑bias questions. Applying these methods to VSI‑Bench, CV‑Bench, and MMMU shows significant gaps between blind and vision‑aware performance and demonstrates that debiased benchmarks can reduce blind scores while preserving vision scores.

By Ellis Brown, Jihan Yang, Shusheng Yang, Rob Fergus, Saining Xie
arXiv Computer Vision
Oct 9

Learning Projection-Aware 360-Degree Image Rectification via Dual-Projection Fusion

The paper introduces a dual‑projection framework for rectifying 360° panoramic images, combining a convolutional neural network that processes equirectangular projections with a vision transformer that handles cubemap projections. Cross‑projection feature transformation and multi‑level fusion allow the two representations to interact, supporting collaborative inclination estimation and upright panorama generation. Experiments on SUN360 and M3D demonstrate significant accuracy gains, achieving 65.9% and 85.2% within a 1° error threshold and low Fréchet Inception Distance scores.

By Yuhao Shan, Qianyi Yuan, Jingguo Liu, Shigang Li, Jianfeng Li, Tong Chen
arXiv Computer Vision
Oct 9

Super-Resolution in The Right Latent Space: A Frozen Vision-Foundation Substrate

The paper proposes that super‑resolution should be performed in a latent space that is close to the manifold of high‑resolution images, rather than in reconstruction‑oriented or pixel spaces. By using the latent space of 23 fused layers of a frozen DINOv3‑L model, degraded images are embedded near this manifold, enabling a 415M decoder (RAESR) to map them back and decode to pixels in a single pass. RAESR achieves the best fidelity–perception trade‑off on several benchmarks while running at 37 ms per 512×512 image on a single H20 GPU, outperforming models that use a VAE latent substrate.

By Wanzhou Lei, Cuifeng Shen, Yanjin He, Maohua Li, Hua Yuan, Per-Olof Persson, Tao Lan, Kan Liu, Hanlin Tang
arXiv Computer Vision
Oct 9

FreeLoc: Online Floorplan Localization via Diffusion-Aided Pose Refinement

FreeLoc is an online RGB-based floorplan localization framework that treats the floorplan as a directly queryable geometric map. It uses efficient online geometric querying and diffusion-aided refinement to retrieve pose anchors and refine them into accurate continuous pose estimates. For sequential localization, it constructs online likelihoods from coarse-sampled candidates and refined hypotheses, enabling histogram-filter-based temporal fusion without offline databases.

By Haocheng Peng, Boyang Zhou, Jiarui Hu, Xiyue Guo, Ziyang Zhang, Boming Zhao, Yifan Gao, Xiao Li, Hujun Bao, Zhaopeng Cui
arXiv AI
Oct 9

UniData: Universal Multimodal Instruction Generation Pipeline

UniData is a universal multimodal instruction generation pipeline that transforms simple user requirements into multi‑round, multimodal instructions. It expands user inputs into diverse events, uses an any‑to‑any large model for instruction generation, and refines the output by correcting irrelevant or redundant inference flows. The authors also introduce UniDataset, a 20,000‑entry dataset spanning nine modalities, and show that UniData achieves state‑of‑the‑art data quality and improves other multimodal models’ understanding and generation capabilities.

By Jiaqi Tang, Yi-Feng Wu, Yuting Zhang, Hao Lu, Bowen Fu, Qing-Guo Chen, Xiaogang Xu, Yuwei Hu, Shiyin Lu, Wei Wei, Lei Zhang, Zhao Xu, Weihua Luo, Qifeng Chen, Ying-Cong Chen
arXiv Machine Learning
Oct 9

Score-Based Learning of Cluster DAGs from Interventions

The paper introduces COARSE, a score‑based method for learning cluster directed acyclic graphs (DAGs) from interventional data. COARSE retains the two‑phase structure of prior constraint‑based approaches—first identifying clusters, then edges—but replaces the edge‑learning phase with a score‑based local search under a cluster‑level BIC score, assuming linear Gaussian relationships. The authors prove polynomial‑time complexity and consistency (given correct intervention targets) and demonstrate that COARSE achieves comparable edge recovery to state‑of‑the‑art methods while being up to two orders of magnitude faster, even on dense graphs with hundreds of nodes.

By Gaetano Tedesco (University of Amsterdam), Alex Markham (University of Copenhagen)
arXiv Machine Learning
Oct 9

A mesh-based neural energy method for the simulation of heterogeneous composites

The paper introduces the mesh-based neural energy method (M‑NEM) for simulating heterogeneous composites, extending the neural energy method (NEM) by discretizing the displacement field on a mesh. By interpolating nodal displacements with shape functions, M‑NEM suppresses displacement oscillations, replaces automatic differentiation with algebraic shape‑function derivatives for strain, and uses high‑order Gaussian quadrature for energy integration. Benchmarks show that M‑NEM reduces stress errors by up to three orders of magnitude and is one to two orders faster than the conventional collocation‑based NEM, while also converging on problems with extreme stiffness contrasts where the conventional method fails.

By Pius J. M. Wichmann, Stefan Hildebrand, Sandra Klinge
arXiv AI
Oct 9

When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection

The paper introduces PARCEL, a benchmark designed to evaluate whether legal claims are supported by cited authorities. Using 3,396 claims from New York State Court of Appeals decisions, the authors label each claim as Supported, Refuted, or Not Found and cast the task as a three‑way natural language inference problem. State‑of‑the‑art LLMs achieve up to 0.97 accuracy in zero‑shot settings, yet they still frequently misclassify unsupported claims as supported, especially when fabricated but plausible citations are present.

By M. Mikail Demir, M. Abdullah Canbaz
arXiv Machine Learning
Oct 9

Selective Listening: Mechanism-Guided Control of Audio Influence in Large Audio-Language Models

The paper introduces ICAP-Gate, a mechanism-guided, task-conditioned control that selectively limits audio influence in large audio-language models (LALMs). By targeting late audio pathways identified through paired drift analysis, ICAP‑Gate reduces influence rate and answer flip across four LALMs and two reasoning benchmarks while preserving automatic speech recognition performance. Compared to fixed suppression and mitigation prompting, ICAP‑Gate offers lower paired‑drift estimates and competitive stabilization with minimal latency overhead.

By Yulin Sun, Kele Xu, Yong Dou
arXiv Computer Vision
Oct 9

Adaptive Adversarial Augmentation for Controllable Face Synthesis

The paper introduces Ensemble Feedback Controllable Synthesis (EFCS), a framework that generates diverse, challenging synthetic face images while preserving visual realism. EFCS improves distributional variability, achieving higher FID and KID scores than single‑feedback or random synthesis, yet maintains high precision. Models trained on EFCS data outperform baselines on multiple benchmarks, and the authors provide an analytical formulation linking perturbation difficulty, sample utility, and performance degradation to guide synthetic data complexity.

By Saransh Suri, Shivang Agarwal, Mayank Vatsa, Richa Singh
arXiv AI
Oct 9

One Skill Too Many: How Co-Installed Skills Conflict in Coding Agents

The paper investigates how co-installed coding‑agent skills can conflict, causing the model to choose a similar skill over the intended one based solely on name and description. An empirical study of 20,947 repositories uncovered 822,109 candidate skill pairs, with 312 confirmed conflicts tested across three models, revealing that nearly one in four installed skills is overridden by a similar skill. Key findings include that conflicts often involve normative skills, that the installed skill loses core functions in about one‑fifth of runs, and that the first skill read determines the outcome, suggesting benchmarks should evaluate exclusive core functions and platforms should guard the initial read.

By Chaoliang Yan, Zihao Xu, Yuekang Li, Shangzhi Xu, Yi Liu, Gelei Deng, Siqi Ma
arXiv AI
Oct 9

Stochastic Grouping Conformal Prediction for Effective Subgroup Reliability

Stochastic Grouping Conformal Prediction (SGCP) is a new framework that provides distribution‑free coverage guarantees while addressing coverage disparities across clinically important subgroups. SGCP learns a stochastic grouping map so that each sample can draw calibration data from others with similar calibration behavior, creating a local score law that improves reliability for all subpopulations. The authors prove that SGCP maintains the standard coverage guarantee and demonstrate through synthetic and real‑world experiments that it reduces subgroup coverage gaps while keeping prediction set sizes small or comparable to existing methods.

By Meihui Zhong, Wenxin Tai, Ting Zhong, Fan Zhou
arXiv AI
Oct 9

When Should Agents Think? Adaptive Reasoning via Cross-Turn Estimation

Large language model agents often reason before every action, but this may be unnecessary if earlier reasoning still supports later actions. The paper introduces RACE, a method that uses likelihood-based signals to detect when reasoning can be omitted, called LoGiC. Experiments on four benchmarks show RACE cuts reasoning cost while keeping or improving performance.

By Yiruo Cheng, Shen Huang, Xiaoshuai Song, Jiejun Tan, Guanting Dong, Pengjun Xie, Ji-Rong Wen, Zhicheng Dou
arXiv AI
Oct 9

Heterogeneous-Modal Unsupervised Domain Adaptation via Latent Space Bridging

The paper introduces Heterogeneous-Modal Unsupervised Domain Adaptation (HMUDA), a setting where a labeled source domain and an unlabeled target domain have entirely different modalities, bridged by an unlabeled domain containing paired observations from both modalities. It proposes Latent Space Bridging (LSB), a dual‑branch framework that uses a feature consistency loss on paired bridge samples to close the modality gap and a class‑centroid alignment loss to reduce source‑target discrepancy. Experiments on eight benchmark settings, covering both 2D‑to‑3D and 3D‑to‑2D transfers, show that LSB achieves state‑of‑the‑art performance.

By Jiawen Yang, Shuhao Chen, Shengtao Zhang, Ke Tang, Yu Zhang
arXiv AI
Oct 9

SAM 3D Animal: Promptable Animal 3D Reconstruction from Images in the Wild

SAM 3D Animal is a promptable framework that performs multi‑animal 3D reconstruction from a single image, using the SMAL+ parametric animal model. It jointly reconstructs multiple instances and allows flexible prompts such as keypoints and masks to better handle crowded and occluded scenes. The authors also introduce Herd3D, a new dataset of over 5,000 images that increases species, interaction, and occlusion diversity, and demonstrate state‑of‑the‑art performance on several benchmark datasets.

By Xuyi Hu, Jin Lyu, Jiuming Liu, Yebin Liu, Silvia Zuffi, Liang An, Stefan Goetz
arXiv Machine Learning
Oct 9

Adaptive Multi-Discriminator WGAN Framework for Resource-Constrained Internet of Vehicles Using Reinforcement Learning and Game Theory

The paper proposes an adaptive multi‑discriminator Wasserstein GAN (MD‑WGAN) framework for Internet of Vehicles (IoV) that combines reinforcement learning and game‑theoretic coordination to manage resource allocation for distributed training. Roadside units host generators with Deep Q‑Network agents that select discriminator subsets and coordinate training epochs, while a unified objective links adversarial quality to resource efficiency, communication overhead, and latency. Experiments on real‑world NGSIM trajectory data show comparable prediction accuracy to state‑of‑the‑art GANs, with a 28% reduction in CPU usage and 6% reduction in memory usage, while maintaining competitive communication overhead and latency.

By Farhoud Jafari Kaleibar, Amr M. Zaki, Marin Litoiu
arXiv Computation and Language
Oct 9

The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection

The paper introduces MARS-Gov, a standpoint-aware multi-agent system designed to detect and mitigate bias in Dutch government documents. It combines legal evidence retrieval, open-set target screening, specialized jurors, conservative routing, and rewrite verification, and adds a dynamic "10th juror" when new bias groups are identified. On the DGDB benchmark, MARS-Gov achieves state‑of‑the‑art performance with an F1 score of 0.880, surpassing zero‑shot LLM detectors and supervised Dutch encoders while reducing unnecessary interventions to 2.5%.

By Yuchen Miao, Zijun Wang, Chang Han, Yurui Shi, Mingtai Zhang, Siyang Xu