Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,507 stories · RSS feed

arXiv AI

Jun 2

MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models

arXiv:2606. 00909v1 Announce Type: cross Abstract: This work presents MLLM-Microscope, a novel system designed for analyzing the hidden representations within Multimodal Large Language Models (MLLMs).

By Ravil Mussabayev, Rustam Mussabayev

llms rag multimodal benchmarks

arXiv AI

Jun 2

CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences

arXiv:2606. 00931v1 Announce Type: cross Abstract: Instruction-guided image editing is becoming a general interface for visual work, yet existing benchmarks still focus largely on narrow appearance edits and do not fully capture the diversity of real-image tasks in professional workflows.

By Fangzhou Lin, Peiran Li, Lingyu Xu, Wenjing Chen, Qianwen Ge, Shuo Xing, Mingyang Wu, Xiangbo Gao, Siyuan Yang, Kazunori Yamada, Ziming Zhang, Haichong Zhang, Zhen Dong, Ming-Hsuan Yang, Zhengzhong Tu

agents computer-vision multimodal benchmarks

arXiv AI

Jun 2

Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks

arXiv:2606. 00920v1 Announce Type: cross Abstract: Run-level pass rate overstates retry-free coverage by up to 17.

By Yongxi Zhou, Lai Yun Choi, Jiaxi Wen, Wenbo Ye

llms benchmarks

arXiv AI

Jun 2

CEAR: Certified Ensemble Adversarial Robustness in DNNs

arXiv:2606. 01437v1 Announce Type: cross Abstract: Deep Neural Networks (DNNs) are highly susceptible to adversarial perturbations, leading to extensive research on robustness for safety-critical applications.

By Daniel Sadig, Mohammadreza Maleki, Hamed Karimi, Reza Samavi

benchmarks safety

arXiv AI

Jun 2

SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision

arXiv:2606. 01139v1 Announce Type: new Abstract: Agent skills are procedural artifacts that enable LLM agents to execute workflows, verify constraints, and recover from failures.

By Yuxuan Liu, Zhaochen Su, Lingyun Xie, Yuhao Zhang, Qing Zong, Jiahe Guo, Zhongwei Xie, Yiyan Ji, Yauwai Yim, Hongyu Luo, Xiyu Ren, Ruan Chenyu, Haoran Li, Yangqiu Song

llms agents benchmarks

arXiv AI

Jun 2

Ethical Fairness in Ubiquitous Health Sensing without Known Attributes

arXiv:2603. 13373v4 Announce Type: replace-cross Abstract: In ubiquitous and mobile health systems, computational models infer human states from wearable, behavioral, and physiological sensing data.

By Shaily Roy, Harshit Sharma, Daniel A. Adler, Srijan Sen, Tanzeem Choudhury, Asif Salekin

benchmarks safety

arXiv AI

Jun 2

Efficient Exploration for Iterative Nash Preference Optimization

arXiv:2606. 01382v1 Announce Type: cross Abstract: Preference alignment is central to improving large language models, but standard reward-based formulations can be restrictive when human preferences are cyclic, non-transitive, or otherwise not representable by a scalar reward.

By Tianlong Nan, Xiaopeng Li, Christian Kroer, Tianyi Lin

llms fine-tuning benchmarks safety

arXiv AI

Jun 2

Shortcut to Nowhere: Demystifying Deep Spurious Regression

arXiv:2606. 01723v1 Announce Type: cross Abstract: Real-world regression often exhibits shortcuts: attributes that are spuriously correlated with continuous targets in training, yet unreliable under deployment shifts; regressing targets using such shortcuts may fail catastrophically at test time.

By Guanrong Xu, Jessica Li, Hao Wang, Yuzhe Yang

llms computer-vision benchmarks

arXiv AI

Jun 2

DOT-MoE: Differentiable Optimal Transport for MoEfication

arXiv:2606. 01666v1 Announce Type: cross Abstract: The scaling of Large Language Models (LLMs) has driven significant performance gains but created substantial challenges in inference efficiency.

By Udbhav Bamba, Arnav Chavan, Aryamaan Thakur, Steve Teig, Deepak Gupta

llms efficiency benchmarks

arXiv AI

Jun 2

Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearning

arXiv:2503. 11832v5 Announce Type: replace Abstract: Recent vision language models (VLMs) have made remarkable strides in generative modeling with multimodal inputs, particularly text and images.

By Yiwei Chen, Yuguang Yao, Yihua Zhang, Bingquan Shen, Gaowen Liu, Sijia Liu

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 2

Why Not Hyperparameter-Friendly Optimisation? A Monotonic Adaptive Norm Rescaling Approach For Long-Tailed Recognition

arXiv:2606. 02526v1 Announce Type: cross Abstract: Long-tailed recognition poses a significant challenge for deep learning.

By Shuo Zhang, Chenqi Li, Tingting Zhu

benchmarks

arXiv AI

Jun 2

VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Models

arXiv:2601. 03309v2 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA) models, which integrate pretrained large Vision-Language Models (VLM) into their policy backbone, are gaining significant attention for their promising generalization capabilities.

By Jianke Zhang, Xiaoyu Chen, Qiuyue Wang, Mingsheng Li, Yanjiang Guo, Yucheng Hu, Jiajun Zhang, Shuai Bai, Junyang Lin, Jianyu Chen

llms computer-vision robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 2

AdaCodec: A Predictive Visual Code for Video MLLMs

arXiv:2606. 02569v1 Announce Type: cross Abstract: Video is temporally redundant: adjacent frames usually share most objects, background, and layout.

By Haowen Hou, Zhen Huang, Zheming Liang, Qingyi Si, Chenglin Li, Shuai Dong, Kele Shao, Ruilin Li, Dianyi Wang, Nan Duan, Jiaqi Wang

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

Dimensionality Reduction for Robust Federated Learning: A Theoretical Analysis and Convergence Guarantee

arXiv:2605. 28335v2 Announce Type: replace Abstract: Federated Learning (FL) enables multiple clients to collaboratively train models without sharing raw data, but it is highly vulnerable to Byzantine attacks.

By Shiyuan Zuo, Jiashuo Li, Rongfei Fan, Han Hu, Jie Xu

rag benchmarks

arXiv AI

Jun 2

Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics

arXiv:2601. 04946v3 Announce Type: replace-cross Abstract: Automatic metrics are widely used to evaluate text-to-image models, often replacing human judgment in benchmarking, model selection, and large-scale data filtering.

By Subhadeep Roy, Gagan Bhatia, Steffen Eger

rag diffusion multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Interventional Processes for Causal Uncertainty Quantification

arXiv:2410. 14483v3 Announce Type: replace-cross Abstract: Reliable uncertainty quantification for causal effects is crucial in high-stakes applications, but remains challenging when the target is an entire function rather than a scalar estimand.

By Hugh Dance, Peter Orbanz, Arthur Gretton

benchmarks

arXiv Machine Learning

Jun 2

Embedding-Space Diffusion for Zero-Shot Environmental Sound Classification

arXiv:2412. 03771v3 Announce Type: replace-cross Abstract: Zero-shot learning enables models to generalise to unseen classes by leveraging semantic information, bridging the gap between training and testing sets with non-overlapping classes.

By Ysobel Sims, Alexandre Mendes, Stephan Chalup

rag diffusion computer-vision benchmarks safety

arXiv Machine Learning

Jun 2

Correcting Gradient-Based Circuit Localization via Interaction-Aware Backpropagation

arXiv:2505. 17630v4 Announce Type: replace-cross Abstract: Circuit localization methods aim to identify the subset of model components responsible for specific behaviors in large language models, enabling detailed mechanistic analysis.

By Joakim Edin, Casper L. Christensen, R\'obert Csord\'as, Tuukka Ruotsalo, Zhengxuan Wu, Maria Maistro, Jing Huang, Lars Maal{\o}e

llms benchmarks safety

arXiv Machine Learning

Jun 2

HR-VILAGE-3K3M: A Human Respiratory Viral Immunization Longitudinal Gene Expression Dataset for Systems Immunity

arXiv:2505. 14725v2 Announce Type: replace-cross Abstract: Respiratory viral infections pose a global health burden, yet the cellular immune mechanisms underlying protection and pathology remain unclear.

By Xuejun Sun, Yiran Song, Xiaochen Zhou, Ruilie Cai, Yu Zhang, Xinyi Li, Rui Peng, Jialiu Xie, Yuanyuan Yan, Muyao Tang, Prem Lakshmanane, Baiming Zou, James S. Hagood, Raymond J. Pickles, Didong Li, Fei Zou, Xiaojing Zheng

benchmarks

arXiv AI

Jun 2

StreamingVLM: Real-Time Understanding for Infinite Video Streams

arXiv:2510. 09608v2 Announce Type: replace-cross Abstract: Vision-language models (VLMs) could power real-time assistants and autonomous agents, but they face a critical challenge: understanding near-infinite video streams without escalating latency and memory usage.

By Ruyi Xu, Guangxuan Xiao, Yukang Chen, Liuning He, Yao Lu, Song Han

llms agents fine-tuning efficiency multimodal benchmarks safety

Get the daily briefing