Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,750 stories · RSS feed

arXiv AI

Jul 7

MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling

arXiv:2602. 17658v3 Announce Type: replace-cross Abstract: Reward modeling is central to alignment pipelines such as RLHF, RLAIF, and PPO-based policy optimization, yet its reliability is constrained by limited and heterogeneous human preference data that are expensive to collect at scale.

By Payel Bhattacharjee, Osvaldo Simeone, Ravi Tandon

reinforcement-learning benchmarks safety

arXiv Machine Learning

Jul 7

Missingness as Signal: Channel-Independent Spectrogram Learning for Clinical Time Series Prediction

arXiv:2607. 02938v1 Announce Type: new Abstract: Clinical time series prediction in intensive care units remains challenging due to heterogeneous physiological variables and informative missingness.

By Soyeon Park, Charmgil Hong

benchmarks

arXiv AI

Jul 7

NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Rooms

arXiv:2606. 20408v3 Announce Type: replace-cross Abstract: Large language model (LLM) agents are increasingly proposed as supervisory components for safety-critical systems, yet their robustness under sustained, adaptive adversarial pressure remains poorly characterized.

By Hanwool Lee, Dasol Choi, Bokyeong Kim, Haon Park, Seung Geun Kim

llms agents benchmarks safety

arXiv AI

Jul 7

IRC-Bench: Recognizing Entities from Contextual Cues in First-Person Reminiscences

arXiv:2605. 06142v2 Announce Type: replace-cross Abstract: When people recount personal memories, they often refer to people, places, and events indirectly, relying on con-textual cues rather than explicit names.

By Yehudit Aperstein, Eden Moran, Alexander Apartsin

llms rag nlp fine-tuning benchmarks

arXiv AI

Jul 7

Incompressible Knowledge Probes: Estimating Black-Box LLM Parameter Counts via Factual Capacity

arXiv:2604. 24827v2 Announce Type: replace-cross Abstract: Closed-source frontier labs do not disclose parameter counts.

By Bojie Li

llms benchmarks safety

arXiv AI

Jul 7

CorridorVLA: Explicit Spatial Constraints for Generative Action Heads via Sparse Anchors

arXiv:2604. 21241v2 Announce Type: replace-cross Abstract: Vision--Language--Action (VLA) models often use intermediate representations to connect multimodal inputs with continuous control, yet spatial guidance is often injected implicitly through latent features.

By Dachong Li, ZhuangZhuang Chen, Jin Zhang, Jianqiang Li

multimodal benchmarks

arXiv AI

Jul 7

StarTSE: Towards Streaming Target Speaker Extraction via Chunk-wise Interleaved Splicing of Autoregressive Language Model

arXiv:2604. 19635v2 Announce Type: replace-cross Abstract: While generative models have set new benchmarks for Target Speaker Extraction (TSE), their inherent reliance on global context precludes deployment in real-time applications.

By Shuhai Peng, Hui Lu, Jinjiang Liu, Liyang Chen, Guiping Zhong, Jiakui Li, Huimeng Wang, Haiyun Li, Liang Cao, Shiyin Kang, Zhiyong Wu

llms benchmarks

arXiv AI

Jul 7

Governed MCP: Kernel-Level Tool Governance for AI Agents via Logit-Based Safety Primitives

arXiv:2604. 16870v2 Announce Type: replace-cross Abstract: AI agents increasingly call external tools (file system, network, APIs) through the Model Context Protocol (MCP).

By Daeyeon Son

llms agents benchmarks safety

arXiv AI

Jul 7

CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas

arXiv:2604. 15267v2 Announce Type: replace-cross Abstract: It is increasingly important that LLM agents interact effectively and safely with other goal-pursuing agents, yet, recent works report the opposite trend: LLMs with stronger reasoning capabilities behave _less_ cooperatively in mixed-motive games such as the prisoner's dilemma and public goods settings.

By Emanuel Tewolde, Xiao Zhang, David Guzman Piedrahita, Vincent Conitzer, Zhijing Jin

llms agents benchmarks safety

arXiv AI

Jul 7

Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching

arXiv:2603. 27044v3 Announce Type: replace-cross Abstract: Deep Reinforcement Learning (DRL) is widely recognized as sample-inefficient, a limitation attributable in part to the high dimensionality and substantial functional redundancy inherent to the policy parameter space.

By Andrea Fraschini, Davide Tenedini, Riccardo Zamboni, Mirco Mutti, Marcello Restelli

diffusion reinforcement-learning benchmarks

arXiv AI

Jul 7

Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents

arXiv:2604. 00392v2 Announce Type: replace-cross Abstract: Agents that synthesize their own tools ship a second artifact alongside each answer: a software library that future tasks reuse, compose, and depend on.

By Alibek Kaliyev, Artem Maryanskyy

llms agents benchmarks safety

arXiv AI

Jul 7

Streaming Model Cascades for Semantic SQL

arXiv:2604. 00660v2 Announce Type: replace-cross Abstract: Modern data warehouses extend SQL with semantic operators that invoke large language models on each qualifying row, making per-row inference orders of magnitude more expensive than traditional SQL.

By Pawe{\l} Liskowski, Kyle Schmaus

llms benchmarks data-engineering

arXiv AI

Jul 7

Multi-Agent Reinforcement Learning for V2X Resource Allocation: Disentangling MARL Challenges Through Benchmarking

arXiv:2603. 06607v2 Announce Type: replace-cross Abstract: Radio resource allocation (RRA) is a critical function in cellular vehicle-to-everything (C-V2X) networks, where vehicles must share limited wireless resources to support safety-critical communications.

By Siyuan Wang, Lei Lei, Pranav Maheshwari, Sam Bellefeuille, Kan Zheng

agents reinforcement-learning benchmarks safety

arXiv AI

Jul 7

When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-On

arXiv:2603. 05659v3 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) and Rubrics as Rewards (RaR) have driven strong gains in domains with clear correctness signals and even in subjective domains by synthesizing evaluation criteria from ideal reference answers.

By Wisdom Ikezogwo, Mehmet Saygin Seyfioglu, Ranjay Krishna, Karim Bouyarmane

reinforcement-learning benchmarks

arXiv AI

Jul 7

Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving

arXiv:2602. 24044v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently.

By Ferran Agullo, Joan Oliveras, Chen Wang, Alberto Gutierrez-Torre, Olivier Tardieu, Alaa Youssef, Jordi Torres, Josep Ll. Berral

llms efficiency benchmarks

arXiv AI

Jul 7

OSF: On Pre-training and Scaling of Sleep Foundation Models

arXiv:2603. 00190v2 Announce Type: replace-cross Abstract: Polysomnography (PSG) provides the gold standard for sleep assessment but suffers from substantial heterogeneity across recording devices and cohorts.

By Zitao Shuai, Zongzhe Xu, David Yang, Wei Wang, Yuzhe Yang

benchmarks

arXiv Machine Learning

Jul 7

Incentivizing Vision Language Models to Search for Long Video Question Answering

arXiv:2607. 02959v1 Announce Type: cross Abstract: We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process.

By Harsh Goel, S P Sharan, Sahil Shah, Minkyu Choi, Joungbin An, Kristen Grauman, Sandeep P. Chinchali

llms agents nlp reinforcement-learning benchmarks

arXiv AI

Jul 7

SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery

arXiv:2602. 17395v2 Announce Type: replace-cross Abstract: Generalized Category Discovery (GCD) aims to identify novel categories in unlabeled data while leveraging a small labeled subset of known classes.

By Lorenzo Caselli, Marco Mistretta, Simone Magistri, Andrew D. Bagdanov

efficiency multimodal benchmarks

arXiv Machine Learning

Jul 7

Piecewise Dynamic Diffusion Regularization for Reconstruction of Cardiac Cine MRI

arXiv:2607. 03299v1 Announce Type: cross Abstract: Real-time cardiac cine MRI enables visualization of the beating heart during free breathing, but severe undersampling and motion make reconstruction highly challenging.

By Florian F\"urnrohr, Reinhard Heckel

diffusion benchmarks

arXiv Machine Learning

Jul 7

TT-Sparse: Learning Sparse Rule Models with Differentiable Truth Tables

arXiv:2603. 07606v2 Announce Type: replace Abstract: Interpretable machine learning is essential in high-stakes domains where decision-making requires accountability, transparency, and trust.

By Hans Farrell Soegeng, Sarthak Ketanbhai Modi, Thomas Peyrin

benchmarks safety

Get the daily briefing