Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,111 stories · RSS feed

arXiv Computation and Language

Oct 27

Of Chains and Gains: What Reinforcement Learning changes on Reasoning-based Image Quality Assessment

arXiv:2610.33203v1 Announce Type: cross Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has improved the performance of large Vision-Language Models (VLMs) on image quality assessment...

By Yixuan Li, Yuxuan Jiang, Hanwei Zhu, Hancheng Zhu, Fan Zhang, Shiqi Wang, David Bull

llms reinforcement-learning multimodal benchmarks

arXiv Computation and Language

Oct 27

PathKGen: Pathology Knowledge Assisted MLLM for Explainable Histopathology

arXiv:2610.33691v1 Announce Type: cross Abstract: Whole Slide Image (WSI) analysis serves as the fundamental method for the clinical diagnosis of cancer in digital histopathology. Recent advances in...

By Nadun Rajapaksha, Sandun Herath, Devnith Wijesinghe, Isiri Amani Withanawasam, Hasindri Watawana, Ranga Rodrigo

llms multimodal benchmarks

arXiv Computation and Language

Oct 27

Humans in Space: Can MLLMs Construct Human-Centric Spatial Understanding via Active Exocentric Exploration?

arXiv:2610.34149v1 Announce Type: cross Abstract: While Multimodal Large Language Models (MLLMs) excel at zero-shot reasoning, their spatial capabilities are primarily evaluated through static, passi...

By Tianchen Guo, Tengyue Zhang, Chen Liu, Ling Chen, Xin Yu

llms fine-tuning multimodal benchmarks

arXiv Computation and Language

Oct 27

Improving Reasoning with Rollout Agreement for Group-Based Policy Optimization

arXiv:2610.34951v1 Announce Type: cross Abstract: Recent methods for Reinforcement Learning with Verifiable Rewards (RLVR) use rollout diversity to improve reasoning performance or training efficienc...

By Sohyun Son, Yerim Hong, Jaeyun Park, Mujeen Sung

reinforcement-learning benchmarks

arXiv Computation and Language

Oct 27

The Voice on the Line

arXiv:2610.35272v1 Announce Type: cross Abstract: A transcriber listening through one microphone also hears everyone near it. We train causal voice isolators: masking transformers of 140M, 243M and 3...

By Husein Zolkepli

llms agents multimodal benchmarks

arXiv Computation and Language

Oct 27

Large Language Model probabilities cannot reliably distinguish between possible and impossible language

arXiv:2509.15114v2 Announce Type: replace Abstract: A controversial test for Large Language Models concerns the ability to discern possible from impossible language. While some evidence attests to th...

By Evelina Leivada, Raquel Montero, Paolo Morosi, Natalia Moskvina, Tamara Serrano, Nikoleta Pantelidou, Marcel Aguilar, Fritz Guenther

llms benchmarks

arXiv Computation and Language

Oct 27

SimulSense: Sense-Driven Interpreting for Efficient Simultaneous Speech Translation

arXiv:2509.21932v3 Announce Type: replace Abstract: How to make human-interpreter-like read/write decisions for simultaneous speech translation (SimulST) systems? Current state-of-the-art systems for...

By Haotian Tan, Hiroki Ouchi, Sakriani Sakti

llms benchmarks

arXiv Computation and Language

Oct 27

Toward Global Large Language Models in Medicine

arXiv:2601.02186v2 Announce Type: replace Abstract: Despite continuous advances in medical technology, the global distribution of health care resources remains uneven. The development of large langua...

By Rui Yang, Huitao Li, Weihao Xuan, Heli Qi, Xin Li, Kunyu Yu, Yingjian Chen, Rongrong Wang, Jacques Behmoaras, Tianxi Cai, Bibhas Chakraborty, Qingyu Chen, Lionel Tim-Ee Cheng, Marie-Louise Damwanza, Chido Dzinotyiwei, Aosong Feng, Chuan Hong, Yusuke Iwasawa, Yuhe Ke, Linah Kitala, Taehoon Ko, Jisan Lee, Irene Li, Jonathan Chong Kai Liew, Hongfang Liu, Lian Leng Low, Edison Marrese-Taylor, Yutaka Matsuo, Isheanesu Misi, Yilin Ning, Jasmine Chiat Ling Ong, Marcus Eng Hock Ong, Enrico Petretto, Hossein Rouhizadeh, Abiram Sandralegar, Oren Schreier, Iain Beehuat Tan, Patrick Tan, Daniel Shu Wei Ting, Junjue Wang, Chunhua Weng, Matthew Yu Heng Wong, Fang Wu, Yunze Xiao, Xuhai Xu, Qingcheng Zeng, Zhuo Zheng, Yifan Peng, Douglas Teodoro, Nan Liu

llms benchmarks

arXiv Computation and Language

Oct 27

OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions

arXiv:2602.05843v3 Announce Type: replace Abstract: The advancement of Large Language Models (LLMs) has catalyzed the development of autonomous agents capable of navigating complex environments. Howe...

By Fangzhi Xu, Hang Yan, Qiushi Sun, Jinyang Wu, Zixian Huang, Muye Huang, Jingyang Gong, Zichen Ding, Kanzhi Cheng, Yian Wang, Xinyu Che, Zeyi Sun, Jian Zhang, Zhangyue Yin, Haoran Luo, Ben Kao, Qika Lin

llms agents benchmarks

arXiv Computation and Language

Oct 27

QUEST: Training Frontier Deep Research Agents with Fully Synthetic Tasks

arXiv:2605.24218v2 Announce Type: replace Abstract: Deep research agents extend the role of search engines from retrieving keyword-matched pages to synthesizing knowledge, fundamentally changing how...

By Jian Xie, Tianhe Lin, Zilu Wang, Yuting Ning, Yuekun Yao, Tianci Xue, Zhehao Zhang, Zhongyang Li, Kai Zhang, Yufan Wu, Shijie Chen, Boyu Gou, Mingzhe Han, Yifei Wang, Vint Lee, Xinpeng Wei, Xiangjun Wang, Yu Su, Huan Sun

agents reinforcement-learning fine-tuning benchmarks

arXiv Computation and Language

Oct 27

Towards Root Memory: Benchmarking and Enhancing Implicit Logical Memory Retrieval for Personalized LLMs

arXiv:2606.23283v2 Announce Type: replace Abstract: Memory systems are essential for personalized Large Language Models (LLMs). However, retrieval within these systems primarily relies on semantic si...

By Hongxun Ding, Xiang Yu, Chengbing Wang, Jianfei Xiao, Keqin Bao, Wenjie Wang, Xiangnan He

llms benchmarks safety

arXiv Computation and Language

Oct 27

Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs

arXiv:2605.17558v2 Announce Type: replace-cross Abstract: Training tool-calling agents requires large-scale trajectory data with verifiable labels, yet existing approaches either synthesize environme...

By Yuxuan Lu, Ziyi Wang, Yingzhou Lu, Yisi Sang, Jiri Gesi, Xianfeng Tang, Yimeng Zhang, Zhenwei Dai, Hui Liu, Hanqing Lu, Chen Luo, Qi He, Benoit Dumoulin, Jing Huang, Dakuo Wang

llms rag agents reinforcement-learning benchmarks

arXiv Computation and Language

Oct 27

"I Strongly Suspect This Website Is a Scam": Benchmarking PII Leakage and Detection without Defense in Autonomous Web Agents

arXiv:2606.00497v2 Announce Type: replace-cross Abstract: Deceptive web content, widely instantiated across the internet and commonly known as \textit{social-engineering attacks}, manipulates autonom...

By Soham Roy, Sarthakbrata Halder, Arya Bharaty, Vaibhav Bhaskar, Yash Sinha, Dhruv Kumar, Srikant Panda, Murari Mandal

llms agents benchmarks safety

arXiv Computation and Language

Oct 27

SearchJev: A Calibrated System-1 Model for Search Agents

arXiv:2610.05107v2 Announce Type: replace-cross Abstract: Search agents repeatedly make short decisions about relevance, evidence sufficiency, and search actions. Using generative language models for...

By Congfeng Cao, Lipeng Zuo, Konstantinos Papakostas, Qiwei Xu, Songwei Xu, Qianru Meng, Lun Zhou, Zhaochun Ren, Yougang Lyu, Xiaohui Yan

llms agents benchmarks

arXiv Computer Vision

Oct 27

Gaussian Episodic Memory for Long-Term Spatiotemporal Reasoning

arXiv:2610.33158v1 Announce Type: new Abstract: Real-world environments undergo continual semi-static changes, requiring a memory representation that captures both the spatial and temporal evolution...

By Ling-Chia Chen, Abdelrhman Werby, Kaan Aytekin, Kai O. Arras

agents nlp robotics benchmarks

arXiv Computer Vision

Oct 27

Compression of Vision Foundation Models for Retinal OCT via Growing-Neural-Gas Codebooks and Graph-Guided Masked Image Modeling

arXiv:2610.33340v1 Announce Type: new Abstract: Vision foundation models for retinal imaging are large and computationally demanding, limiting point-of-care deployment. We ask how much of such a mode...

By Ali Yavari, Tilman Schmoll, Rainer A. Leitgeb, Wolfgang Drexler, Thomas Schlegl

llms computer-vision benchmarks

arXiv Computer Vision

Oct 27

Bidirectional Cross-frequency Domain Interaction Network for UHD Low-Light Image Enhancement

arXiv:2610.33402v1 Announce Type: new Abstract: Existing Ultra-High-Definition (UHD) low-light image enhancement (LLIE) methods typically rely on either spatial- or frequency-domain modeling, or inde...

By Bo Fu, Hai Chi, Liyan Wang, Shilin Fu, Wei Wang, Kin-Man Lam, Jinshan Pan, Cong Wang

benchmarks

arXiv Computer Vision

Oct 27

AV-FABLE: Audio-Visual Manipulation Forensics in Long-Form Videos

arXiv:2610.33413v1 Announce Type: new Abstract: Localized AI manipulations in long-form videos often affect brief events within otherwise authentic content, making video-level classification insuffic...

By Yuesheng Huang, Hoi Fan Au, Jiaqi Hou, Ruicheng Zhang, Jiarui He, Haocheng Yang, Bin Hu, Kewei Zhang, Weijian Deng, Jun Zhou, Xiu Li

llms robotics multimodal benchmarks

arXiv Computer Vision

Oct 27

Cross-Scale Sparse Attention for Referring Segmentation in Remote Sensing

arXiv:2610.33551v1 Announce Type: new Abstract: High-resolution remote sensing (RS) imagery provides fine spatial detail over large areas, making it essential for fine-grained scene understanding. Re...

By Yuxi Li, Lu Si, Xia Liao, Yidong Shen, Zheng Wei, Jun Zhang, Xin Yuan

computer-vision efficiency multimodal benchmarks safety

arXiv AI

Oct 27

AgentOrch: How Close Can Global Orchestration Come to Step-by-Step Interaction?

AgentOrch is a benchmark that compares global orchestration—where a complete workflow is generated upfront—with step‑by‑step interaction on 650 enterprise standard operating procedures and 3,220 runtime variants. Across all evaluated models, step‑by‑step remains more accurate, but the performance gap has narrowed by about 79% from early to recent models. The study identifies three methods—incremental workflow construction, revising workflows with execution feedback, and targeted training on complete workflows—that can further close the gap, improving global orchestration accuracy by up to 12.18 percentage points.

By Zijie Liu, Qi Yang Huang, Jinhao Duan, Yueh Che Lee, Thomas A. Joseph, Zayne Turner, Adam Seligman, Tianlong Chen, Zhaozhuo Xu

llms agents benchmarks

Get the daily briefing