Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv AI

Oct 26

PEAR: Adaptive Advantage Reshaping for Perception and Exploration in Large Vision-Language Models

PEAR is an adaptive advantage reshaping method designed to improve perception and exploration in large vision‑language models. It builds on visual dependency as a base signal and adds an entropy‑based exploration gain only at high‑entropy positions, giving larger gains where visual dependency is lower. Experiments on Qwen2.5‑VL‑3B/7B demonstrate that PEAR outperforms baseline methods across multimodal reasoning benchmarks and achieves leading results on textual reasoning and general multimodal tasks.

By Junxin Wang, Yingfeng Luo, Xiangchen Song, Weiqiao Shan, Runsong Zhao, Xinyu Liu, Shunjie Xing, Yifu Huo, Dingyang Lin, Qiuyu Gu, Yongyu Mu, Chenglong Wang, Changliang Li, Jingbo Zhu, Tong Xiao

llms multimodal benchmarks

arXiv AI

Oct 26

From Visual Affinity to Learning Credit in Multimodal Reinforcement Learning

The paper introduces Outcome-Conditioned Visual Credit (OVC), a plug‑in framework for multimodal reinforcement learning that separates visual association measurement from token‑level credit assignment. OVC uses Relative Visual Encoding to create a context‑relative visual coordinate and Outcome‑Conditioned Evidence Calibration to adjust token contributions based on outcome‑discriminative evidence. When applied to GSPO-token, OVC improves mean accuracy by 1.86–2.39 percentage points across six benchmarks without requiring additional reward models or inference‑time modules.

By Yuhang Han, Dingning Liu, Yujie Chen, Yaojie Zhang, Jingwei Song, Zequn Yang, Xuanming Shang, Li Ding, Zhijiang Guo, Linfeng Zhang

reinforcement-learning multimodal benchmarks

arXiv AI

Oct 26

FOCUS: A Lightweight Mamba-2 Model for EEG-Based Cognitive Workload Classification

FOCUS is a lightweight Mamba‑2-based model designed for EEG‑based binary cognitive workload classification. It encodes multi‑channel EEG with a fixed FFT covering the five canonical bands and uses depthwise‑separable convolution combined with a weight‑shared encoder‑bottleneck Mamba‑2 pair, resulting in only 0.04 million parameters and 0.11 GFLOPs. In a pilot study with 11 participants, FOCUS achieved 74.75 % accuracy and 0.08 ms GPU forward latency, with weight sharing improving accuracy by 4.55 percentage points without statistical significance.

By Ioannis Kyprakis, Stefanos Gkikas, Chariklia Chatzaki, Zongyi Jiang, Randy Gomez, Raul Fernandez Rojas

benchmarks safety

arXiv AI

Oct 26

An Agentic Approach with Evidence-Grounded Verification for Multimodal Emotion Reasoning in Conversation

The paper proposes an agentic framework for multimodal emotion reasoning that uses frozen models to avoid costly instruction tuning. It employs modality‑specific perception agents, an integration agent, and an MLLM critic to generate emotion predictions with evidence citations, followed by an evidence‑grounded verification stage that identifies errors and applies targeted prompt patches without updating model weights. On the MER2025 dataset, this approach improves F‑score across several large language‑vision models, outperforming static pipelines and matching or exceeding reflection baselines.

By Elyas Rashno, Farhana Zulkernine

llms agents fine-tuning multimodal benchmarks

arXiv AI

Oct 26

AutoMIL: Automated Multiple Instance Learning for Whole-Slide Image Classification

AutoMIL is an open‑source framework that automates key steps of multiple instance learning pipelines for whole‑slide image classification, including preprocessing, training, and evaluation. In benchmark tests, it achieved ROC‑AUC scores of 0.94 on UBC‑OCEAN, 0.99 on PCam, and 0.93‑0.96 on a TCGA‑NSCLC subset, matching the performance of specialized pipelines. The framework’s ensembling approach particularly improved results on imbalanced datasets, demonstrating the need for automated pipelines to adapt to dataset characteristics.

By Jonas Waibel, Frank Kramer, Dominik M\"uller

computer-vision benchmarks

arXiv AI

Oct 26

ArtifactArena: Evaluating Models by What They Build in the Physical World

arXiv:2610.06511v2 Announce Type: replace-cross Abstract: To evaluate the frontier, we must measure models not by what they say, but by what they can engineer and build in grounded physical environme...

By Kushagra Tiwary, David Mayo, Nikhil Behari, Xiangzhou Sun, Abdulrahman Alabdulkareem, Isaac Galatzer-Levy, Boris Katz, Brian Cheung

robotics benchmarks

arXiv AI

Oct 26

Learning from Next-Frame Prediction: Towards Unified and Scalable Visual World Modeling

arXiv:2610.30724v2 Announce Type: replace-cross Abstract: Video provides a natural source of supervision: what happens next serves as a learning signal for visual world modeling. Yet masked video pre...

By Yifeng Yao, Jinghan Li, Zhihao Zhang, Yajiao Xiong, Yang Jin, Kun Xu, Huishuai Zhang, Dongyan Zhao

rag nlp benchmarks