Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,257 stories · RSS feed

arXiv AI

Jul 8

MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understanding

arXiv:2605. 30794v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated significant achievements in general visual question answering (VQA) tasks.

By Qian Kou, Xiaofeng Shi, Yulin Li, Xiaosong Qiu, Xinyang Wang, Hua Zhou, Cao Dongxing

llms nlp multimodal benchmarks

arXiv Machine Learning

Jul 8

Fine-Tuning Regimes Define Distinct Continual Learning Problems

arXiv:2604. 21927v3 Announce Type: replace Abstract: Continual learning (CL) studies how models acquire tasks sequentially while retaining previously learned knowledge.

By Paul-Tiberiu Iordache, Elena Burceanu

fine-tuning benchmarks

arXiv AI

Jul 8

PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNet

arXiv:2607. 06097v1 Announce Type: cross Abstract: 3D dense captioning, an emerging vision-language task, aims to generate descriptive sentences for each object in the 3D scene.

By Xiaopei Wu, Chenshu Hou, Liang Peng, Dan Xu, Binbin Lin, Xiaoshui Huang, Yuenan Hou, Yu Li, Wenxiao Wang, Haifeng Liu, Deng Cai, Wanli Ouyang

multimodal benchmarks

arXiv AI

Jul 8

RPAM: A Principled Metric for Evaluating Associations in Language Models with High Predictive Validity in Downstream Outputs

arXiv:2607. 05679v1 Announce Type: cross Abstract: Language models (LMs) exhibit problematic biases, such as stereotypes.

By Damian Hodel, Jevin West, Aylin Caliskan

llms rag benchmarks

arXiv AI

Jul 8

Do Quantum Transformers Help? A Systematic VQC Architecture Comparison on Tabular Benchmarks

arXiv:2604. 23931v2 Announce Type: replace-cross Abstract: Variational quantum circuits (VQCs) are a leading approach to quantum machine learning on near-term devices, yet it remains unclear which circuit architecture yields the best accuracy-parameter trade-off on classical tabular data.

By Chi-Sheng Chen, En-Jui Kuo

llms benchmarks

arXiv AI

Jul 8

MambaGaze: Bidirectional Mamba with Explicit Missing Data Modeling for Cognitive Load Assessment from Eye-Gaze Tracking Data

arXiv:2605. 22775v2 Announce Type: replace-cross Abstract: Real-time cognitive load assessment from eye-tracking signals could enable adaptive human-centered AI in safety-critical applications such as driver vigilance monitoring or automated flight deck assistance, yet two challenges persist: handling frequent data missingness from blinks and tracking failures, and efficiently modeling long-range temporal dependencies.

By Amir Mousavi, Mohammad Sadegh Sirjani, Erfan Nourbakhsh, Mimi Xie, Rocky Slavin, Leslie Neely, John Davis, John Quarles

efficiency benchmarks safety

arXiv AI

Jul 8

BaFCo: A Document Understanding Benchmark for Complex Bangla Form Comprehension

arXiv:2607. 05614v1 Announce Type: cross Abstract: Document comprehension is a challenging yet impactful task for Multimodal Large Language Models, especially as these systems see growing adoption in real-world, human-centric applications.

By Abu Tyeb Azad, Ishita Sur Apan, Fahim Ahmed, Sumaiya Karim Katha, Ezharuddin Jubaer, Armun Alam, Pranjal Kumar Nandi, Amin Ahsan Ali, Aman Chadha, Md Mofijul Islam, AKM Mahbubur Rahman

llms multimodal benchmarks

arXiv AI

Jul 8

Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

arXiv:2607. 06309v1 Announce Type: cross Abstract: Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities.

By Aysan Ghayouri Pirsoltan, Shima Babakordi, Mohammad Reza Mohammadi

llms computer-vision benchmarks

arXiv Machine Learning

Jul 8

EntroPath: Maximum Entropy Path Ensemble Embedding for Manifold Learning

arXiv:2607. 06497v1 Announce Type: new Abstract: We introduce EntroPath, a manifold learning method that recovers geodesic geometry from data graphs through ensembles of diffusion paths.

By Przemys{\l}aw Rola

rag diffusion benchmarks

arXiv Machine Learning

Jul 8

A semantic mutation metric for metamorphic relation adequacy in scientific computing programs

arXiv:2605. 17437v2 Announce Type: replace-cross Abstract: Context.

By Meng Li (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China), Xiaohua Yang (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China), Jie Liu (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China), Shiyu Yan (School of Computing, University of South China, Hengyang, China, Hunan Engineering Research Center of Software Evaluation and Testing for Intellectual Equipment, Hengyang, China, CNNC Key Laboratory on High Trusted Computing, Hengyang, China)

llms benchmarks

arXiv Machine Learning

Jul 8

Learning to Throw Objects Safely in Multi-Obstacle Environments

arXiv:2607. 06388v1 Announce Type: cross Abstract: Robotic throwing enables fast and efficient object placement beyond the robot's immediate workspace, but reliable throwing in cluttered environments remains underexplored.

By Mohammadreza Kasaei, Klemen Voncina, Hamidreza Kasaei

reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jul 8

Fine-Tuning Integrity for Modern Neural Networks: Structured Drift Proofs via Norm, Rank, and Sparsity Certificates

arXiv:2604. 04738v2 Announce Type: replace-cross Abstract: Fine-tuning is the dominant paradigm for adapting large machine learning models, yet current deployment pipelines provide no way to verify how a released model was updated.

By Zhenhang Shang, Yingzhe Yu, Kani Chen

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 8

To Retain or to Adapt? Generalizing Continual Learning

arXiv:2607. 05609v1 Announce Type: cross Abstract: The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting.

By Giulia Lanzillotta, Mandana Samiei, Doina Precup, Razvan Pascanu, Claire Vernade

computer-vision reinforcement-learning benchmarks safety

arXiv AI

Jul 8

Breaking Structural Isolation: Scalable Graph Clustering via Community-Aware Sampling and Structural Entropy

arXiv:2607. 05469v1 Announce Type: cross Abstract: Unsupervised graph clustering is a fundamental technique for uncovering underlying semantic patterns in large-scale networks.

By Jingyun Zhang, Hao Peng, Jianxin Li, Angsheng Li, Philip S. Yu

benchmarks

arXiv AI

Jul 8

FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference

arXiv:2607. 06519v1 Announce Type: new Abstract: Long-context LLM inference is increasingly limited by the memory and bandwidth cost of KV caches, yet aggressive compression can remove the layer-specific evidence needed for retrieval and multi-step reasoning.

By Anna C\'ordoba, Adam Puente Tercero, Nerea Angulo Hijo, Mar Linares Tercero, Julia Barrientos, Ainhoa Miranda, Jes\'us Olivera

llms nlp efficiency benchmarks

arXiv AI

Jul 8

x-Prediction Is All You Need:Training-Free Accelerated Generation via Endpoint Decodability

arXiv:2607. 06114v1 Announce Type: cross Abstract: Diffusion and flow matching models generate high-quality samples, but their ODE samplers often need tens to hundreds of neural function evaluations (NFEs).

By Xin Peng, Ang Gao

diffusion efficiency benchmarks

arXiv AI

Jul 8

Most LLM Conformity Needs No Speaker: Measuring the Speaker-Free Floor in Peer-Pressure Benchmarks

arXiv:2607. 05545v1 Announce Type: cross Abstract: LLM conformity is often used to describe cases where a model changes a correct answer toward a peer or group response.

By Yibo Hu, Jiaming Qu

llms benchmarks

arXiv AI

Jul 8

PatchOptic for Shared-State LLM Workflows with Projected Views and Verified Structured Updates

arXiv:2607. 05483v1 Announce Type: cross Abstract: Agentic workflows often operate over shared, structured state.

By Zhaoyu Bai, Jiaqi Cai

llms rag agents benchmarks

arXiv Machine Learning

Jul 8

PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies

arXiv:2602. 19710v3 Announce Type: replace-cross Abstract: Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency because they entangle high-level perception with sparse, embodiment-specific action supervision.

By Haitao Lin, Hanyang Yu, Jingshun Huang, He Zhang, Yonggen Ling, Ping Tan, Xiangyang Xue, Yanwei Fu

nlp robotics multimodal benchmarks safety

arXiv AI

Jul 8

FORGE: Towards Functional Tool-Use Generalization via Keypoint Trajectory Reasoning

arXiv:2607. 05780v1 Announce Type: cross Abstract: While humans readily repurpose a book, a stone, or a shoe to drive a nail, robots trained on specific tools fail to transfer the same function to novel ones -- a gap we formalize as functional generalization.

By Chuhao Zhou, Liquan Wang, Shuxin Cao, Xiangyu Chen, Yuxuan Hu, Boyu Ma, Animesh Garg, Jianfei Yang

robotics benchmarks

Get the daily briefing