Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,494 stories · RSS feed

arXiv AI

Jun 3

ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning

arXiv:2606. 03503v1 Announce Type: new Abstract: Large Reasoning Models (LRMs) have achieved remarkable progress thanks to Reinforcement Learning with Verifiable Rewards (RLVR) on Chain-of-Thoughts (CoTs).

By Ziyan Liu, Xueda Shen, Yuzhe Gu, Songyang Gao, Kuikun Liu, Guangran Cheng, Chengqi Lyu, Dahua Lin, Wenwei Zhang, Kai Chen

reinforcement-learning benchmarks

arXiv AI

Jun 3

Adaptive Latent Agentic Reasoning

arXiv:2606. 02871v1 Announce Type: cross Abstract: Large reasoning models improve performance by generating extended chain-of-thought (CoT) reasoning, but this behavior becomes inefficient when applied to LLM agents.

By Dongwon Jung, Peng Shi, Yi Zhang, Junshan Zhang, Muhao Chen

llms agents benchmarks

arXiv AI

Jun 3

Hallucinations as Orthogonal Noise: Inference-Time Manifold Alignment via Dynamic Contextual Orthogonalization

arXiv:2606. 03022v1 Announce Type: cross Abstract: Hallucination in Large Language Models (LLMs), characterized by the generation of content inconsistent with contextual facts or logical constraints -- remains a persistent challenge for reliable deployment.

By Mingkuan Zhao, Wentao Hu, Tianchen Huang, Yuheng Min, Suquan Chen, Yide Gao, Yanbo Zhai, Shuangyong Song, Xuelong Li

llms benchmarks safety

arXiv AI

Jun 3

Towards Compact Autonomous Driving Perception with Balanced Learning and Multi-sensor Fusion

arXiv:2606. 02979v1 Announce Type: cross Abstract: We present a novel compact deep multi-task learning model to handle various autonomous driving perception tasks in one forward pass.

By Oskar Natan, Jun Miura

agents computer-vision robotics benchmarks

arXiv AI

Jun 3

Do Neural Retrievers Prefer Certain Documents? Evidence of Learned Relevance Priors

arXiv:2606. 02814v1 Announce Type: cross Abstract: Neural retrievers are trained to estimate query-document relevance from annotated query-document pairs.

By Francisco Valentini, Edgar Altszyler, Martin Fajcik

llms rag benchmarks safety

arXiv AI

Jun 3

Plan, Verify and Fill: A Structured Parallel Decoding Approach for Diffusion Language Models

arXiv:2601. 12247v3 Announce Type: replace-cross Abstract: Diffusion Language Models (DLMs) present a promising non-sequential paradigm for text generation, distinct from standard autoregressive (AR) approaches.

By Miao Li, Hanyang Jiang, Sikai Cheng, Hengyu Fu, Yuhang Cai, Baihe Huang, Tinghan Ye, Xuanzhou Chen, Pascal Van Hentenryck

llms diffusion benchmarks

arXiv AI

Jun 3

AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following

arXiv:2606. 03116v1 Announce Type: cross Abstract: The rapid advancement of instruction-guided audio generation has highlighted the critical need for robust alignment evaluation.

By Haitao Li, Tian Tan, Yuguang Yang, Shan Yang, Xie Chen

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 3

When Should LLMs Be Less Specific? Selective Abstraction for Reliable Long-Form Text Generation

arXiv:2602. 11908v3 Announce Type: replace Abstract: LLMs are widely used, yet they remain prone to factual errors that erode user trust and limit adoption in high-risk settings.

By Shani Goren, Ido Galil, Ran El-Yaniv

llms benchmarks

arXiv AI

Jun 3

Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approach

arXiv:2510. 23216v4 Announce Type: replace Abstract: While several high profile video games have served as testbeds for Deep Reinforcement Learning (DRL), this technique has rarely been employed by the game industry for crafting authentic AI behaviors.

By Alessandro Sestini, Joakim Bergdahl, Jean-Philippe Barrette-LaPierre, Florian Fuchs, Brady Chen, Fabio Zinno, Michael Jones, Linus Gissl\'en

agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 3

Toward Training Superintelligent Software Agents through Self-Play SWE-RL

arXiv:2512. 18552v3 Announce Type: replace-cross Abstract: While current software agents powered by large language models (LLMs) and agentic reinforcement learning (RL) can boost programmer productivity, their training data (e.

By Yuxiang Wei, Zhiqing Sun, Emily McMilin, Jonas Gehring, David Zhang, Gabriel Synnaeve, Daniel Fried, Lingming Zhang, Sida Wang

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 3

Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition

arXiv:2606. 03657v1 Announce Type: new Abstract: Large language models for code generation often need to use APIs that are absent from their pretraining data.

By Jinnuo Liu, Yue Peng, Jinhan Niu, Hongyi Wen

llms agents fine-tuning benchmarks

arXiv AI

Jun 3

The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios

arXiv:2601. 08173v2 Announce Type: replace Abstract: The rapid evolution of Multi-modal Large Language Models (MLLMs) has advanced workflow automation; however, existing research mainly targets performance upper bounds in static environments, overlooking robustness for stochastic real-world deployment.

By Daocheng Fu, Jianbiao Mei, Rong Wu, Xuemeng Yang, Jia Xu, Ding Wang, Pinlong Cai, Yong Liu, Licheng Wen, Botian Shi

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jun 3

WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization

arXiv:2603. 04956v2 Announce Type: replace Abstract: This paper considers the problem of converting a given dense linear layer to low precision.

By Egor Lifar, Semyon Savkin, Or Ordentlich, Yury Polyanskiy

llms efficiency benchmarks

arXiv AI

Jun 3

What Makes Interaction Trajectories Effective for Training Terminal Agents?

arXiv:2606. 03461v1 Announce Type: new Abstract: Stronger code agents are commonly assumed to be superior teachers for post-training, yet this assumption remains poorly disentangled from task difficulty, harness design, and student capacity.

By Sidi Yang, Chaofan Tao, Jierun Chen, Tiezheng Yu, Ruoyu Wang, Yuxin Jiang, Yiming Du, Wendong Xu, Jing Xiong, Taiqiang Wu, Lifeng Shang, Xiaohui Li, Ngai Wong, Haoli Bai

llms agents fine-tuning benchmarks

arXiv AI

Jun 3

InftyThink+: Effective and Efficient Infinite-Horizon Reasoning via Reinforcement Learning

arXiv:2602. 06960v3 Announce Type: replace-cross Abstract: Large reasoning models achieve strong performance by scaling inference-time chain-of-thought, but this paradigm suffers from quadratic cost, context length limits, and degraded reasoning due to lost-in-the-middle effects.

By Yuchen Yan, Liang Jiang, Jin Jiang, Shuaicheng Li, Zujie Wen, Zhiqiang Zhang, Jun Zhou, Jian Shao, Yueting Zhuang, Yongliang Shen

nlp reinforcement-learning efficiency benchmarks

arXiv AI

Jun 3

PnP-Corrector: A Universal Correction Framework for Coupled Spatiotemporal Forecasting

arXiv:2605. 08935v3 Announce Type: replace Abstract: Coupled spatiotemporal forecasting is important for predicting the future evolution of multiple interacting dynamical systems, such as in climate models.

By Hao Wu, Fan Xu, Yuxu Lu, Penghao Zhao, Fan Zhang, Hao Jia, Yuxuan Liang, Ruijian Gou, Qingsong Wen, Xian Wu, Xiaomeng Huang, Yuan Gao

agents benchmarks

arXiv Machine Learning

Jun 3

Parallel Adaptive Multi-Objective Evolutionary Learning of Discretized Bayesian Network Classifiers for Clinical Data

arXiv:2605. 29058v2 Announce Type: replace Abstract: Bayesian Networks (BNs) are of interest from an explainable AI viewpoint, offering transparent probabilistic models for decision support.

By Damy M. F. Ha, Thalea Schlender, Yvette M. van der Linden, Peter A. N. Bosman, Tanja Alderliesten

benchmarks

arXiv AI

Jun 3

Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphs

arXiv:2606. 03705v1 Announce Type: new Abstract: Knowledge Graphs (KGs) are widely used to mitigate the limitations of Large Language Models (LLMs), such as outdated knowledge and hallucinations.

By Weiwei Ding, Zixuan Li, Long Bai, Zhuo Chen, Kun Su, Fei Wang, Xiaolong Jin, Jin Zhang, Jiafeng Guo, Xueqi Cheng

llms benchmarks safety

arXiv AI

Jun 3

Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregation

arXiv:2606. 03128v1 Announce Type: cross Abstract: Smart contracts face critical security challenges that require thorough auditing in decentralized web services.

By Bagus Rakadyanto Oktavianto Putra, Muhamad Risqi Utama Saputra, Widyawan, Guntur Dharma Putra

llms efficiency benchmarks safety

arXiv AI

Jun 3

Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents

arXiv:2605. 08747v4 Announce Type: replace Abstract: Standard embodied evaluations do not independently score whether an agent correctly commits to task completion at episode closure, a capacity we call terminal commitment.

By Ying Chen, Lihuang Fang, Rui Jiang, Mingxu Wang, Zhifeng Gu, Lei Yi, Jie Chen

agents robotics benchmarks

Get the daily briefing