Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,127 stories · RSS feed

arXiv Machine Learning

Oct 15

Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards

arXiv:2610.19372v1 Announce Type: new Abstract: Runtime policy verifiers can prevent agents from executing disallowed actions, but their decisions are typically used only for enforcement and discarde...

By Min-Young Yu

agents benchmarks

arXiv AI

Oct 15

KV4-RL: Unlocking Faster Asynchronous Rollouts for RLVR with INT4 KV Cache Quantization

arXiv:2610.14448v1 Announce Type: cross Abstract: Asynchronous rollout improves the efficiency of reinforcement learning with verifiable rewards (RLVR) by reducing under-utilization during long-tail...

By Sihwan Park, Sung-Yub Kim, Doohyuk Jang, Eunho Yang

reinforcement-learning efficiency benchmarks

arXiv AI

Oct 15

EV-JEPA: Elastic V-JEPA for Any Device and Cross-Device Inference

arXiv:2610.14324v1 Announce Type: cross Abstract: Foundation models like V-JEPA are increasingly deployed across a heterogeneous edge-cloud continuum spanning smart glasses, phones, robots, and serve...

By Anxhelo Diko, Jieyu Lin, Stefano Ambrogio, Ziyun Li, Mahmoud Assran, Th\'eo Moutakanni, Barbara De Salvo, Chiao Liu

robotics benchmarks

arXiv AI

Oct 15

Releasing the Inner Jev: A Simple Recipe for Open Decision Models

arXiv:2610.17205v1 Announce Type: new Abstract: LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a ty...

By Jonathan Geuter, Andrew Cheng, David Alvarez-Melis

llms fine-tuning benchmarks

arXiv AI

Oct 15

VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

arXiv:2610.17057v1 Announce Type: new Abstract: Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages. This l...

By Simon Ott, Yufan Chen, Ruiping Liu, Junwei Zheng, Jiale Wei, Di Wen, Kunyu Peng, Jiaming Zhang, Rainer Stiefelhagen

rag benchmarks

arXiv Machine Learning

Oct 15

FOLD: Position-Free Far Memory with Local Order for Long-Context Speculative Decoding

arXiv:2610.20392v1 Announce Type: cross Abstract: Speculative decoding accelerates LLM generation through lightweight drafting and parallel verification. It is particularly promising for long inputs,...

By Shuxuan Li, Jialiang Kang, Yanghao Zhou, Ziye Zhu, Xiao Yang

llms benchmarks

arXiv Computer Vision

Oct 15

VGGT-Diff: Visual Geometry Meets Diffusion for Sparse-View Novel View Synthesis

arXiv:2609.33253v2 Announce Type: replace Abstract: We present VGGT-Diff, a geometry-routed multi-view diffusion model for sparse-view novel view synthesis. Existing novel view synthesis (NVS) method...

By Kangjie Chen, Xiangyu Li, Dongbin Zhang, Chaoda Zheng, Shijia Chen, Jinhao Deng, Hongbin Lin, Choo Sin Wai, Minqi Wang, Minghao Yang, Dake Zhong, Guorui Song, Yu Zhang, Xianming Liu, Boyang Wang

diffusion benchmarks

arXiv AI

Oct 15

Structural Chain-of-Thought: Reasoning with Executable Computational Graphs

arXiv:2610.15107v1 Announce Type: new Abstract: Chain-of-thought prompting enables reasoning in large language models but produces explanations that are difficult to verify, expensive to generate, an...

By Petr Senatorov, Karim Vafin, Elena Tutubalina, Oleg Somov

llms reinforcement-learning benchmarks

arXiv AI

Oct 15

PsyMemBehav: Psychology-Informed and Memory-Grounded Two-Level Identity-to-Action Reasoning for Social-Media Behavior Simulation

arXiv:2610.14570v1 Announce Type: new Abstract: Large language models (LLMs) support social simulation, but faithfully reproducing real users requires grounding both structured platform decisions and...

By Wenji Zhang, Xilong Cheng, Yunxiao Qin, Mai Tu, Tongye Zhu, Yuhao Zhou, JingBo Ji, Yuting Tan, Duoyin Li, Hongjiang Xiao, Yuan Zhang

llms reinforcement-learning benchmarks

arXiv AI

Oct 15

RoboHarness: A Simple Harness Outperforms VLA and World Actions Models

arXiv:2610.14018v1 Announce Type: cross Abstract: The rise of large language models (LLMs) has opened a path toward generalist embodied AI, raising the question of how LLMs can act in the physical wo...

By Bince Qu, Wei Chen, Bo Zhang

llms agents robotics multimodal benchmarks

arXiv AI

Oct 15

Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference

arXiv:2610.14277v1 Announce Type: new Abstract: Tensor parallelism ends every layer with an all-reduce that sits between two dependent matrix multiplications, so its cost cannot be hidden and grows a...

By Xingyu Xie, Xiaoyong Xie, Zhouchen Lin, Kim-Chuan Toh

llms efficiency benchmarks

arXiv AI

Oct 15

Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation

arXiv:2409.00696v4 Announce Type: replace-cross Abstract: Rating-based human evaluation has become an essential tool to accurately evaluate the impressive performance of large language models (LLMs)....

By Jasper Dekoninck, Maximilian Baader, Martin Vechev

llms benchmarks safety

arXiv AI

Oct 15

Horizon Scans can be accelerated using novel information retrieval and artificial intelligence tools

arXiv:2504.01627v2 Announce Type: replace-cross Abstract: Introduction: Horizon scanning in healthcare assesses early signals of innovation, crucial for timely adoption. Current horizon scanning face...

By Lena Schmidt, Oshin Sharma, Chris Marshall, Sonia Garcia Gonzalez Moral

llms rag benchmarks

arXiv AI

Oct 15

Efficient Multi-Agent Reasoning via Confidence-Guided Adaptive Debate

arXiv:2610.16343v1 Announce Type: new Abstract: Multi-agent debate has shown promise for improving the reasoning of large language models, yet recent theory suggests its benefits are highly regime-de...

By Seungdong Yoa, Ye Seul Sim, Suhee Yoon, Sanghyu Yoon, Dongmin Kim, Soonyoung Lee, Bumsoo Kim, Junhyun Lee

llms agents benchmarks

arXiv AI

Oct 15

GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models

arXiv:2506.10946v3 Announce Type: replace-cross Abstract: Unlearning large language models (LLMs) is important due to copyright and privacy concerns. A key challenge is unintended forgetting, where r...

By Peizhi Niu, Evelyn Ma, Boya Hou, Huiting Zhou, Duo Zhou, Huan Zhang, S. Rasoul Etesami, Olgica Milenkovic

llms benchmarks safety

arXiv AI

Oct 15

Naga: Vedic Encoding for Deep State Space Models

arXiv:2511.13510v2 Announce Type: replace-cross Abstract: This paper presents Naga, a deep State Space Model (SSM) encoding approach inspired by structural concepts from Vedic mathematics. The propos...

By Melanie Schaller, Nick Janssen, Bodo Rosenhahn

benchmarks

arXiv AI

Oct 15

MARL-BattleGrounds: A Novel JAX-native Benchmark for Heterogeneous and Competitive Multi-agent RL

arXiv:2610.14883v1 Announce Type: new Abstract: We present MARL-BattleGrounds (MARL-BGs), an easy-to-use JAX-native benchmark for heterogeneous and competitive multi-agent reinforcement learning (MAR...

By Ulixes Tariq Hawili, Subramanian Ramamoorthy, Yoann Altmann, Ignacio Carlucho

llms agents reinforcement-learning benchmarks

arXiv AI

Oct 15

CB-Agent: Concept-Based In-Context Learning for Sequential Action Selection

arXiv:2610.14574v1 Announce Type: new Abstract: In-context learning (ICL) allows large language models to use demonstrations without updating their parameters. For interactive agents, demonstrations...

By Qingyue Zhang, Huaze Tang, Minyu Zhang, Yuxiang Yan, Wenbo Ding, Shao-Lun Huang

llms agents benchmarks

arXiv AI

Oct 15

AcrossVAM2.0: Conflict-Aware Point Supervision for Parameter-Efficient Video Diffusion Adaptation

arXiv:2610.14693v1 Announce Type: new Abstract: Explicit point correspondence is attractive for adapting video generators: it describes how visible surfaces move, but can overwrite the appearance pri...

By Nan Wu, Yee Hung Hong

llms diffusion fine-tuning benchmarks

arXiv AI

Oct 15

SAGE: Pushing 2-Bit Vector-Quantized LLMs toward Real-World Usability

arXiv:2610.14745v1 Announce Type: new Abstract: Extreme low-bit weight-only quantization is an effective approach to reducing the deployment costs of large language models. Below 3 bits, scalar quant...

By Xianglong Yan, Zenan Wu, Kewei Zhang, Kaicheng Yang, Yulun Zhang

llms agents efficiency benchmarks

Get the daily briefing