Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv AI
Oct 9

Examining Social Attribution in LLM Reasoning: A Theory-Guided Probing Methodology

The paper introduces a systematic study of how large language models (LLMs) perform social attribution—specifically responsibility and blame judgments—by creating a benchmark with 7,639 questions drawn from classic attribution theory scenarios and real-world narratives. It evaluates 32 LLMs and 5 non-LLM baselines, finding that larger models show greater but still incomplete alignment with human judgments. A probing methodology further reveals that certain attribution dimensions can be decoded from specific hidden-state positions, and their influence on LLM outputs mirrors patterns predicted by human Attribution Theory.

By Zhaoxin Yu, Qingchao Kong, Dajun Zeng, Wenji Mao
arXiv AI
Oct 9

All Verdicts are Not Equal: Rethinking LLM Judge Reliability

The paper audits the reliability of large language models used as judges in NLP evaluation. It stresses that verdicts can vary across identical replications, change with presentation order, and that even the most deterministic model only agrees with ground truth 51% of the time. The authors introduce the trustworthy verdict rate (T) to quantify reproducibility, order‑invariance, and accuracy, and show that holistic rubric scoring improves trustworthiness more than any single prompting strategy.

By Vineet Kumar, Darshita Rathore, Anindya Moitra
arXiv AI
Oct 9

DVD: Dynamic Vector Decoding for Efficient MLLM-based Perception

The paper introduces DVD, a dynamic vector decoding method that unifies 2D and 3D perception tasks for multimodal large language models. DVD transforms various perceptual representations—such as 2D bounding boxes, masks, and 3D bounding boxes—into 1D vector sequences, maps them to compact discrete tokens, and then decodes them back to the original formats via a lightweight de-tokenizer. Experiments on RefCOCO, SUN‑RGBD, KITTI, Hypersim, and nuScenes show that DVD outperforms existing methods while significantly reducing token overhead and inference latency.

By Jinghua Hou, Zhe Liu, Hengshuang Zhao
arXiv AI
Oct 9

LiteGUI: Lightweight GUI Agents via Multi-Solution Guided Distillation and Dual-Level Reinforcement Learning

LiteGUI is a new framework for training lightweight GUI agents that tackles the long‑horizon, multi‑path nature of GUI interactions. It introduces a two‑stage post‑training pipeline: Guided On‑Policy Distillation, which uses human‑verified multi‑solution action annotations to provide privileged teacher guidance at each state, and Multi‑Solution Dual‑Level GRPO, which combines action‑level supervision with history‑conditioned planning‑quality supervision while handling multiple valid actions. The authors also present a scalable data generation pipeline and the Lite‑Dataset, enabling extensive evaluation across GUI benchmarks and demonstrating substantial performance gains over existing training paradigms.

By Yubin Wu, Zicheng Cai, Liping Ning, Hua Wang, Zhi Chen, Yaohua Tang, Hao Chen
arXiv AI
Oct 9

The Review Lottery: Benchmarking an Observational Estimator of Peer-Review Noise (ICLR 2017-2025)

The paper introduces an observational estimator to quantify how much peer‑review decisions would change if a paper were reviewed by a different set of reviewers, using public review data from ICLR 2017‑2025. The estimator decomposes scores into paper quality and reviewer noise via a Bayesian ordered‑probit model, maps scores to decisions with a logistic model, and simulates independent committees to estimate disagreement rates, finding 23‑30% disagreement at committee size two and 18‑24% at size four. Calibration against NeurIPS 2021 data shows close agreement, and longitudinal analysis reveals no robust trend in reviewer noise over the period, though specific years (2020, 2021) exhibit higher flip rates due to scale compression and low signal‑to‑noise. whyItMatters":"The study provides a scalable, data‑driven method to assess the reliability of conference accept/reject decisions, offering insights into reviewer noise and potential improvements in review processes."

By Feilian Huang
arXiv AI
Oct 9

Learning to Orchestrate Evolutionary Search: Progression-Aware Deep Reinforcement Learning for Dynamic DE-CMA-ES Coordination in Optimization and Structural Model Updating

The paper introduces DRL-DCO, a Deep Reinforcement Learning‑governed dynamic orchestration of Differential Evolution and CMA‑ES for high‑dimensional structural model updating. A DDPG actor‑critic agent uses a progression‑aware state and diversity‑informed reward to continuously reallocate resources between exploration and exploitation, while controlling population size, elite preservation, and restarts. The trained actor can operate in a supervision‑free inference mode, enabling fast deployment with maintained performance, and demonstrates superior convergence accuracy and robustness on benchmark optimization and structural health monitoring problems.

By Lechen Li (State Key Laboratory of Internet of Things for Smart City, University of Macau, Macau 519000, China, College of Water Conservancy and Hydropower Engineering, Hohai University, Nanjing 210098, China), Rongye Shi (School of Artificial Intelligence, Beihang University, Beijing 100191, China), Wanhuan Zhou (State Key Laboratory of Internet of Things for Smart City, University of Macau, Macau 519000, China)
arXiv Machine Learning
Oct 9

V-CoLA: Vision Token Compression with Linear Attention

The paper introduces V-CoLA, a training‑free token compression framework tailored for vision-language models that use linear attention. V-CoLA employs a uniqueness‑aware importance criterion to select critical vision tokens and an adaptive token merging strategy to compress them, all optimized to fit the chunk‑wise parallelism of linear attention. Experiments show that V‑CoLA retains 99.5% of the original performance with only half the vision tokens and achieves significant speedups during prefill.

By Hao Jiang, Yiru Mao, Tianpeng Bu, Hao Zhou, Hongtao Duan, Wang Jing, Bowen Xu, Xin Chen, Lulu Hu, Bin Yang, Yongliang Tao, Minying Zhang
arXiv AI
Oct 9

Conversational Task Disambiguation over Tabular Data: Leakage-Aware Formulation, Benchmark Suite, and Training

The paper introduces a leakage‑aware framework for conversational task disambiguation over tabular data, defining ambiguous verifiable tasks and separating the agent into an asking policy and a solution policy. It presents AmbiTab, a benchmark suite that unifies six ambiguous datasets under a common representation, enabling evaluation of clarification strategies and oracle leakage. The authors train an asking policy with reinforcement learning, achieving improved disambiguation metrics across all datasets and higher task success on five of them, while providing diagnostics to measure oracle leakage during training.

By Nafiseh Ghoroghchian, Luis Scoccola, Tina Sedaghat, Omid Vaheb, Hannah Chen, Dino D'Agostino, Keyvan Golestan
arXiv AI
Oct 9

AliO: Output Alignment Matters in Long-Term Time Series Forecasing

AliO: Output Alignment Matters in Long-Term Time Series Forecasting proposes a new method, AliO, to improve output alignment in LTSF models by reducing discrepancies in prediction outputs for the same timestamps across both time and frequency domains. The authors introduce a Time Alignment Metric (TAM) to quantify alignment, contrasting with traditional MSE metrics that only measure distance to ground truth. Experiments demonstrate that AliO can increase TAM by up to 58.2% while maintaining or improving forecasting performance by up to 27.5%.

By Kwangryeol Park, Jaeho Kim, Seulki Lee
arXiv Machine Learning
Oct 9

Omni-Diffusion-Distill: Few-Step Distillation of Unified Multimodal Diffusion Large Language Models

Omni‑Diffusion‑Distill is a two‑stage distillation framework that compresses unified multimodal diffusion large language models (dLLMs) into a single efficient student while preserving both image generation and multimodal understanding. In the first stage the student learns to skip decoding steps by replaying cached teacher trajectories, and in the second stage it is refined on intermediate states along its own rollouts. The method introduces a pairwise collision penalty to reduce repetition in parallel text decoding and entropy‑matched guidance to prevent entropy collapse in image generation, achieving state‑of‑the‑art trade‑offs: reducing image generation from 128 to 8 steps and multimodal understanding from 512 to 64 steps, yielding 18.2× and 21.2× wall‑clock speedups and strong performance on GenEval, DPG‑Bench, MM‑Vet, and COCO captioning.

By Hong Huang, Chenhongyi Yang, Junzhe Sun, Animesh Sinha, Wuyang Chen, Yifan Jiang
arXiv AI
Oct 9

Trust, Don't Trust, or Flip: Robust Preference-Based Reinforcement Learning with Multi-Expert Feedback

The paper introduces TriTrust-PBRL (TTP), a framework for preference-based reinforcement learning that learns a shared reward model and expert-specific trust parameters from multi-expert feedback. Trust parameters evolve to be positive (trust), near zero (ignore), or negative (flip), allowing the model to invert adversarial preferences instead of discarding them. Experiments on manipulation and locomotion tasks show TTP achieves near-oracle performance under adversarial corruption, outperforming existing PBRL methods.

By Seyed Amir Hosseini, Maryam Abdolali, Amirhosein Tavakkoli, Fardin Ayar, Ehsan Javanmardi, Manabu Tsukada, Mahdi Javanmardi
arXiv Machine Learning
Oct 9

NEMORA: Neural Equivariant Multipole Operators for Long-Range Atomistic Learning

The paper introduces NEMORA, a neural equivariant extension of the Fast Multipole Method designed to capture long‑range tensorial interactions in atomistic machine learning models. NEMORA replaces the FMM’s fixed multipole expansion with learnable equivariant operators that operate on an adaptive spatial hierarchy, enabling many‑body interactions across multiple length scales while preserving physical inductive biases. The method achieves linear time and memory complexity, allowing it to handle systems with hundreds of thousands of atoms and significantly reducing force and energy errors on non‑local benchmarks compared to short‑range backbones.

By Jay L. Kaplan, Samuel Varner, Rebecca Willett, Juan J. de Pablo
arXiv Machine Learning
Oct 9

Strategic Investment Decision Making for Value Creation in Energy Transition: A Reinforcement Learning Approach

The paper presents a reinforcement learning framework for strategic investment decisions in the energy transition, aiming to balance profit, CO2 social costs, and competitive advantage across oil & gas, renewables, and CO2 reduction sectors. A custom simulation environment models the dynamic energy landscape up to 2050, allowing the RL agent to learn optimal policies that adapt to uncertainties in production, prices, and costs. Benchmarking shows the RL strategy consistently outperforms manually defined baseline policies in adaptability and long-term value creation.

By Yasaman Cheraghi (Department of Energy and Petroleum Engineering, University of Stavanger, Norway), Reidar B. Bratvold (Department of Energy and Petroleum Engineering, University of Stavanger, Norway), Aojie Hong (Independent Researcher, Stavanger, Norway), Ressi B. Muhammad (Department of Energy and Petroleum Engineering, University of Stavanger, Norway), Sergey Alyaev (NORCE Norwegian Research Centre, Bergen, Norway)
arXiv Machine Learning
Oct 9

3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs

The paper introduces 3BASiL-TM, a one‑shot post‑training method for compressing large language models using a sparse plus low‑rank decomposition. It employs a novel 3‑block ADMM algorithm to minimize layer‑wise reconstruction error and a transformer‑matching refinement that jointly optimizes sparse and low‑rank components across layers. Experiments show a >30% reduction in WikiText2 perplexity gap for LLaMA‑8B under a (2:4 sparse + 64 LR) setting and a 2.5× faster compression runtime on an A100 GPU compared to state‑of‑the‑art methods.

By Mehdi Makni, Xiang Meng, Rahul Mazumder
arXiv Computer Vision
Oct 9

DiscoVL: Unveiling Disentangled C ross-Modal Representation Learning via Orthogonal Adversarial Regularization for V ision-Language Models

DiscoVL introduces a disentangled cross‑modal representation learning framework for vision‑language models, combining orthogonal adversarial regularization with a multi‑branch low‑rank residual aligner to enhance bidirectional visual‑textual interaction. The method decomposes representations into subspaces, enabling cross‑modal feedback at each layer and preventing centroid collapse through an orthogonal adversarial triplet loss. Across 15 benchmarks, DiscoVL consistently outperforms state‑of‑the‑art approaches in base‑to‑novel generalization, cross‑dataset evaluation, and few‑shot learning.

By Mengping Dong, Jinbao Li, Fei Li
arXiv Computer Vision
Oct 9

IntactWorld: Joint World Modeling with Intact Features

IntactWorld introduces a Joint World Modeling Architecture that preserves uncompressed Intact Features to better capture real‑world logic in video generation. By predicting clean intermediate features instead of flow velocity in a high‑dimensional space, it avoids manifold gaps and reduces computational overhead through a Full‑to‑Compact Training Paradigm that replaces raw features with refined CLS tokens. The approach cuts spatial memory usage by 11.4% and inference latency by 43.8%, achieving a 2.46‑point improvement on the VBench 2.0 benchmark.

By Boming Tan, Xiangdong Zhang, Yan Xia, Qi Zhu, Deyi Ji, Xue Yang, Shaofeng Zhang
arXiv Computer Vision
Oct 9

WorldFact-Bench: Beyond Image-Internal Plausibility to Image-World Consistency

WorldFact-Bench is a new benchmark that tests image‑world consistency by presenting 1,274 real‑fake image pairs across four verification regimes and ten semantic domains, each pair containing a factual conflict while keeping the image visually plausible. The evaluation requires both images in a pair to be classified correctly, and the benchmark focuses on single‑image assessment without predefined claims. The authors also introduce PERSIST‑Agent, an iterative verification system that maintains a persistent state linking candidate facts, visual observations, evidence, and verification statuses, and shows improved pair accuracy over baseline methods on 8B backbones.

By Zhuohong Chen, Zhengxian Wu, Yunyao Yu, Hangrui Xu, Zijian Yu, Hao Tan, Zhifang Liu, Peng Jiao, Jun Lan, Haoqian Wang
arXiv Computer Vision
Oct 9

Fast Pose Tracking of Rigid Objects with Compact Pose Graph Optimization

The paper presents a lightweight 6D pose tracking method that operates on relative pose constraints instead of point-based optimization, using uncertainty derived from geometric alignment to weight these constraints. By avoiding direct inclusion of noisy point measurements, the approach achieves fast and robust long-term tracking with computational costs far lower than reconstruction-based trackers. Experiments on four real-world benchmarks show comparable accuracy to state-of-the-art methods while maintaining real-time performance.

By Xiaojie Zhang, Tom Fischer, Viktor Larsson, Eddy Ilg
arXiv Computer Vision
Oct 9

IS-Diff: Improving Diffusion-Based Inpainting with Better Initial Seed

The paper introduces IS‑Diff, a training‑free diffusion‑based inpainting method that improves semantic consistency by using initial seeds sampled from unmasked regions. It incorporates a dynamic selective refinement mechanism to adjust the initialization strength during intermediate latent stages. Experiments on CelebA‑HQ, ImageNet, and Places2 show that IS‑Diff outperforms state‑of‑the‑art methods across standard and large‑mask inpainting tasks.

By Yongzhe Lyu, Yu Wu, Yutian Lin, Bo Du
arXiv AI
Oct 9

Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems

The paper introduces EMFA, a method for attributing failures in large language model-based multi-agent systems by identifying the decisive error—an agent–step pair whose correction would restore correct execution. EMFA builds a structured representation of the failed trajectory, models error propagation and persistent loops, and applies propagation-aware screening with counterfactual verification to pinpoint the decisive error. On the Who&When benchmark, EMFA outperforms prior methods, improving step-level attribution accuracy by 3.45 and 4.40 percentage points on two subsets.

By Jiaqi Liao, Yuanzhao Zhai, Huanxi Liu, Xu Zhang, Zheming Zhuang, Dawei Feng, Bo Ding, Huaimin Wang