Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI

Oct 5

Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation

arXiv:2610.02324v1 Announce Type: cross Abstract: Foundation multimodal large language models are designed to support a broad spectrum of capabilities across diverse domains. Multi-teacher on-policy...

By Xiaofei Yin, Tong Chu, Jiyuan Fu, Jun Lan, Shuheng Zhou, Huijia Zhu

llms efficiency multimodal benchmarks

arXiv AI

Oct 5

DeskForge: Dense Supervision from Desktop Environments for Computer-Use Agents

arXiv:2610.02320v1 Announce Type: cross Abstract: Computer-use agents need to reliably ground action targets in complex desktop scenes, where multiple applications, overlapping windows, and visually...

By A. Said Gurbuz, Ahmed Nassar, Sunghwan Hong, Marc Pollefeys, Peter W. J. Staar

llms agents fine-tuning multimodal benchmarks

arXiv AI

Oct 5

EXAM2: Extending Audio Understanding in Multilingual and Multimodal Analysis

arXiv:2608.23758v3 Announce Type: replace-cross Abstract: Recent large audio language models (LALMs) have achieved impressive progress in audio understanding. However, existing evaluations remain lar...

By Jiawen Wang, Xiaoxue Gao, Zi Haur Pang, Nancy F. Chen

llms fine-tuning multimodal benchmarks

arXiv Computer Vision

Oct 5

Transform-Aligned Learned Features for Lossy Point Cloud Attribute Compression

arXiv:2609.34834v2 Announce Type: replace Abstract: Transform-based methods provide an effective framework for point cloud attribute compression by representing attributes as transform coefficients....

By Yueru Chen, Pengpeng Yu, Dingquan Li, Wei Gao, Wei Zhang, Fei Song

efficiency benchmarks

arXiv AI

Oct 5

Counterfactual Predictions in Scientific Emulators Without Controlled Experiments

arXiv:2610.02252v1 Announce Type: cross Abstract: Many scientific questions require reasoning about what was never observed: What if the conditions, interventions, or history had been different? Mode...

By Dingling Yao, Kahaan Gandhi, Valentin Duruisseaux, Boris Bonev, Francesco Locatello, Anima Anandkumar

diffusion benchmarks

arXiv AI

Oct 5

Credit Where It Matters: Dependency-Aware Policy Optimization for Terminal Agents

arXiv:2610.03634v1 Announce Type: new Abstract: Terminal-using agents benefit from reinforcement learning (RL) in coding, debugging, and other multi-step terminal tasks. In these tasks, later command...

By Yu Li, Guangfeng Cai, Long-Fei Li, Shuo Han, Shengtian Yang, Han Luo, Kaibing Yang, Lei Feng

agents reinforcement-learning benchmarks

arXiv AI

Oct 5

MIRROR: Multipath Quorum Integrity for LLM Multi-Agent Communication

arXiv:2610.02349v1 Announce Type: cross Abstract: Inter-agent communication is central to Large Language Model Multi-Agent Systems (LLM-MAS), but it introduces an underexplored vulnerability: Agent-i...

By Ryuichi Yamafuji Lun, Jingzhen Wang, Shreyas Kolte, Ruiteng Li

llms agents multimodal benchmarks

arXiv AI

Oct 5

SPEAR: A Spectral-Disentangled MoE Neural Operator with Knowledge-Guided Expert Aggregation for Large-Scale PDE Pretraining

arXiv:2610.03265v1 Announce Type: cross Abstract: Large-scale pre-training has improved the generalization of neural operators across diverse PDEs. However, existing PDE foundation models still strug...

By Dengdi Sun, Xiaoya Zhou, Xiao Wang, Wanli Lyu, Jin Tang, Bin Luo

fine-tuning benchmarks

arXiv Machine Learning

Oct 5

State-Space Unlearning for Non-Stationary Bias in Land Surface Forecasting

arXiv:2610.02248v1 Announce Type: new Abstract: Operational land surface forecasting systems built on Mamba-family Structured State Space Models absorb non-stationary confounding events (unrecorded i...

By Anidipta Pal

benchmarks safety

arXiv Machine Learning

Oct 5

Hybrid Machine Learning-Assisted Raman Spectroscopy with Generative Feature Augmentation for Pharmaceutical Identification

arXiv:2610.02224v1 Announce Type: new Abstract: Rapid and reliable identification of pharmaceutical residues is important for safeguarding public health, ensuring food safety, and enabling practical...

By Quach Thi Thai Binh, Ton Nu Quynh Trang, Thang B. Phan, Vu Thi Hanh Thu, Nguyen Tuan Hung

rag diffusion benchmarks safety

arXiv AI

Oct 5

Safe and Robust Neural Policy Learning with Statistical Verification for Sim-to-Real Deployment in Robotics

arXiv:2608.06481v2 Announce Type: replace-cross Abstract: Synthesizing safe and robust neural controllers in simulation for reliable sim-to-real deployment remains a critical challenge in robotics. E...

By Riccardo Curcio, Hongpeng Cao, Marco Caccamo

robotics benchmarks safety

arXiv AI

Oct 5

EchoDistill: Robust Large Audio Language Models via Noisy-to-Clean Self-Distillation

arXiv:2605.23954v2 Announce Type: replace-cross Abstract: Large Audio Language Models (LALMs) remain vulnerable to acoustic noise, which can obscure task-relevant evidence and produce unreliable resp...

By Kaiwen Luo, Chunxi Luo, Liang Lin, Yuxuan Li, Zhenhong Zhou, Junhao Dong, Yingjie Zhou, Zhendong Chu

llms efficiency benchmarks

arXiv AI

Oct 5

LEAF: A Living Benchmark for Event-Augmented Forecasting

arXiv:2605.16358v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly applied to real-world forecasting tasks, yet evaluating their true predictive capability remain...

By Mingtian Tan, Mihir Parmar, Palash Goyal, Chun-Liang Li, Nanyun Peng, Thomas Hartvigsen, Jinsung Yoon, Tomas Pfister

llms agents benchmarks

arXiv AI

Oct 5

Evaluation format, not model capability, drives measured triage failure in the assessment of consumer health AI

arXiv:2603.11413v4 Announce Type: replace-cross Abstract: A recent Nature Medicine study reported that ChatGPT Health under-triages 51.6% of emergencies and concluded that consumer-facing AI triage p...

By David Fraile Navarro, Jialei Sheng, Farah Magrabi, Enrico Coiera

llms benchmarks safety

arXiv AI

Oct 5

$T^5$: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training

arXiv:2609.32791v2 Announce Type: replace Abstract: Reinforcement mid-training lets language models learn internal thoughts from unlabeled text, but efficient token-level credit assignment remains ch...

By Nan Qiao, Yebin Yang, Weinong Wang, Shuning Wang, Shangpin Peng, Fengyuan Lu, Xinming Wang, Zhehan Kan, Ruixu Zhang, Songyang Zhang, Sheng Yue, Yonglong Tian, Ju Ren

llms reinforcement-learning benchmarks

arXiv Machine Learning

Oct 5

Understanding Trajectory Heterogeneity in Federated World Model Learning

arXiv:2610.02957v1 Announce Type: new Abstract: World models learn state evolution from trajectories, making access to temporal context a central training requirement. Federated learning can use dist...

By Yipan Wei, Zhaokun Yan, Ziming Hong, Jiaqi Wu, Lixu Wang

benchmarks

arXiv Machine Learning

Oct 5

Learning from Evolving Errors: Adaptive Iterative Repair for On-Policy Distillation

arXiv:2610.02700v1 Announce Type: new Abstract: On-policy self-distillation (OPSD) supplies dense token-level feedback on trajectories sampled from the student's own policy, a richer training signal...

By Rui Li, Liyang He, Zheng Zhang, Zhenya Huang, Linbo Zhu, Qi Liu

reinforcement-learning efficiency benchmarks

arXiv Machine Learning

Oct 5

Teaching LLMs to See Graphs: Unifying Text and Structural Reasoning

arXiv:2605.10247v2 Announce Type: replace Abstract: Applying Large Language Models (LLMs) to graph-structured data usually involves multi-step pipelines in which textual node attributes are compresse...

By Dario Vajda

llms fine-tuning benchmarks

arXiv Machine Learning

Oct 5

From Patching to Pruning Visual Computation in Vision Language Models

arXiv:2610.03389v1 Announce Type: cross Abstract: Vision language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every...

By Rahul Chowdhury, Timothy A Rupprecht, Xuan Shen, Shaoyi Huang, Pu Zhao, Yanzhi Wang

llms efficiency multimodal benchmarks safety

arXiv Machine Learning

Oct 5

Most-Recent Anchoring with Recurrent Ordering for Time Series Forecasting

arXiv:2610.03494v1 Announce Type: new Abstract: Long-term forecasting models commonly process all patches in a look-back window using the same fixed stack. Older contextual patches and recent evidenc...

By Jung Min Choi, Ngoc Son Le, Ibram Abdelmalak, Vijaya Krishna Yalavarthi, Lars Schmidt-Thieme

benchmarks

Get the daily briefing