Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,750 stories · RSS feed

arXiv AI

Jul 7

Agent vs. Parametric World Models: Hybrid Planning for Reliable Language Agents

arXiv:2606. 27806v3 Announce Type: replace Abstract: Language agents plan by generating not only actions but also implicit predictions of how the world will change.

By Xinyuan Song, Zekun Cai

llms agents benchmarks

arXiv Machine Learning

Jul 7

Robustness Meets Uncertainty: Evidential Adversarial Training for Robust Selective Classification

arXiv:2607. 03075v1 Announce Type: new Abstract: Safety-critical applications require classifiers that are both robust and reliable.

By Nicolas Sournac, Ahmed Baha Ben Jmaa, Bertrand Braeckeveldt

benchmarks safety

arXiv Machine Learning

Jul 7

Do ECG Foundation Models Transfer to Rare Cardiac Diseases? Evidence from Brugada Syndrome Detection

arXiv:2607. 03009v1 Announce Type: new Abstract: Background: Foundation models (FMs) trained on large-scale unlabeled physiological data have emerged as a promising paradigm for medical artificial intelligence.

By Beatrice Zanchi, Giuliana Monachino, Alvise Dei Rossi, Luigi Fiorillo, Georgia Sarquella-Brugada, Giulio Conte, Francesca Dalia Faraci

fine-tuning benchmarks safety

arXiv AI

Jul 7

Agentic and Generative AI for Open-Source Intelligence and Cyber Investigations: Taxonomy, Evaluation, Challenges, and Future Directions

arXiv:2607. 03233v1 Announce Type: cross Abstract: The rapid growth of publicly available digital information has rendered manual open-source intelligence (OSINT) analysis insufficient for modern intelligence, cybersecurity, and cyber investigation.

By Eduardo Almeida Palmieri, Mohamed Chahine Ghanem, Dipo Dunsin, Zubair Baig, Ed de Quincey, Kim-Kwang Raymond Choo

llms rag agents nlp fine-tuning multimodal benchmarks safety

arXiv Machine Learning

Jul 7

Missingness as Signal: Channel-Independent Spectrogram Learning for Clinical Time Series Prediction

arXiv:2607. 02938v1 Announce Type: new Abstract: Clinical time series prediction in intensive care units remains challenging due to heterogeneous physiological variables and informative missingness.

By Soyeon Park, Charmgil Hong

benchmarks

arXiv AI

Jul 7

DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections

arXiv:2508. 12116v2 Announce Type: replace-cross Abstract: As numerous instruction-tuning datasets continue to emerge, dynamically balancing and optimizing their mixtures has become a critical challenge.

By Haebin Shin, Lei Ji, Xiao Liu, Zhiwei Yu, Hyunwoo Yoo, Qi Chen, Yeyun Gong

reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 7

PAST-TIDE: Prototype-Anchored Statement Tuning with Topic-Invariant Normalization for Stance Detection

arXiv:2607. 04690v1 Announce Type: cross Abstract: We introduce PAST-TIDE, our stance detection system addressing both subtasks of the StanceNakba Shared Task at NakbaNLP@LREC-COLING 2026.

By Md. Shakhoyat Rahman Shujon, MD Jahid Hasan Jim, Md. Milon Islam, Md Rezwanul Haque, Fakhri Karray

benchmarks

arXiv AI

Jul 7

LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL

arXiv:2607. 04412v1 Announce Type: new Abstract: Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals.

By Yujin Kim, Namgyu Ho, Sangmin Hwang, Joonkee Kim, Yongjin Yang, Sangmin Bae, Seungone Kim, Jaehun Jung, Se-Young Yun, Hwanjun Song

llms reinforcement-learning benchmarks

arXiv AI

Jul 7

CoopEval: Benchmarking Cooperation-Sustaining Mechanisms and LLM Agents in Social Dilemmas

arXiv:2604. 15267v2 Announce Type: replace-cross Abstract: It is increasingly important that LLM agents interact effectively and safely with other goal-pursuing agents, yet, recent works report the opposite trend: LLMs with stronger reasoning capabilities behave _less_ cooperatively in mixed-motive games such as the prisoner's dilemma and public goods settings.

By Emanuel Tewolde, Xiao Zhang, David Guzman Piedrahita, Vincent Conitzer, Zhijing Jin

llms agents benchmarks safety

arXiv AI

Jul 7

Programming over Thinking: Efficient and Robust Multi-Constraint Planning

arXiv:2601. 09097v3 Announce Type: replace Abstract: Multi-constraint planning involves identifying, evaluating, and refining candidate plans while satisfying multiple, potentially conflicting constraints.

By Derrick Goh Xin Deik, Quanyu Long, Zhengyuan Liu, Nancy F. Chen, Wenya Wang

llms benchmarks

arXiv AI

Jul 7

Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching

arXiv:2603. 27044v3 Announce Type: replace-cross Abstract: Deep Reinforcement Learning (DRL) is widely recognized as sample-inefficient, a limitation attributable in part to the high dimensionality and substantial functional redundancy inherent to the policy parameter space.

By Andrea Fraschini, Davide Tenedini, Riccardo Zamboni, Mirco Mutti, Marcello Restelli

diffusion reinforcement-learning benchmarks

arXiv AI

Jul 7

Beyond Task Completion: A Verification-vs.-Conformance Gap in Tool-Evolving Agents

arXiv:2604. 00392v2 Announce Type: replace-cross Abstract: Agents that synthesize their own tools ship a second artifact alongside each answer: a software library that future tasks reuse, compose, and depend on.

By Alibek Kaliyev, Artem Maryanskyy

llms agents benchmarks safety

arXiv AI

Jul 7

Streaming Model Cascades for Semantic SQL

arXiv:2604. 00660v2 Announce Type: replace-cross Abstract: Modern data warehouses extend SQL with semantic operators that invoke large language models on each qualifying row, making per-row inference orders of magnitude more expensive than traditional SQL.

By Pawe{\l} Liskowski, Kyle Schmaus

llms benchmarks data-engineering

arXiv AI

Jul 7

Multi-Agent Reinforcement Learning for V2X Resource Allocation: Disentangling MARL Challenges Through Benchmarking

arXiv:2603. 06607v2 Announce Type: replace-cross Abstract: Radio resource allocation (RRA) is a critical function in cellular vehicle-to-everything (C-V2X) networks, where vehicles must share limited wireless resources to support safety-critical communications.

By Siyuan Wang, Lei Lei, Pranav Maheshwari, Sam Bellefeuille, Kan Zheng

agents reinforcement-learning benchmarks safety

arXiv AI

Jul 7

When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-On

arXiv:2603. 05659v3 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) and Rubrics as Rewards (RaR) have driven strong gains in domains with clear correctness signals and even in subjective domains by synthesizing evaluation criteria from ideal reference answers.

By Wisdom Ikezogwo, Mehmet Saygin Seyfioglu, Ranjay Krishna, Karim Bouyarmane

reinforcement-learning benchmarks

arXiv AI

Jul 7

Data Driven Optimization of GPU efficiency for Distributed LLM-Adapter Serving

arXiv:2602. 24044v2 Announce Type: replace-cross Abstract: Large Language Model (LLM) adapters enable low-cost model specialization, but introduce complex caching and scheduling challenges in distributed serving systems where hundreds of adapters must be hosted concurrently.

By Ferran Agullo, Joan Oliveras, Chen Wang, Alberto Gutierrez-Torre, Olivier Tardieu, Alaa Youssef, Jordi Torres, Josep Ll. Berral

llms efficiency benchmarks

arXiv AI

Jul 7

Spectral Signatures of Large Language Models

arXiv:2607. 03377v1 Announce Type: cross Abstract: The rapidly growing repository of publicly available large language models (LLMs) presents significant challenges for systematic management and quantification at scale, such as model lineage tracing, licensing, and evaluation.

By Zhuoying Zhang, Ishan V. Prasad, Yuanzhe Hu, Zihang Liu, Hengrui Luo, Pu Ren, Yaoqing Yang

llms benchmarks

arXiv AI

Jul 7

OSF: On Pre-training and Scaling of Sleep Foundation Models

arXiv:2603. 00190v2 Announce Type: replace-cross Abstract: Polysomnography (PSG) provides the gold standard for sleep assessment but suffers from substantial heterogeneity across recording devices and cohorts.

By Zitao Shuai, Zongzhe Xu, David Yang, Wei Wang, Yuzhe Yang

benchmarks

arXiv AI

Jul 7

Advanced Topic Modeling Techniques for Categorizing Software Vulnerabilities

arXiv:2607. 03887v1 Announce Type: cross Abstract: The increasing complexity and frequency of software vulnerabilities demand efficient methods to analyze and prioritize threats.

By Utkarsh Tiwari, Spoorthi M, Anirudh S, Nidhin Prabhakar T. V

llms benchmarks

arXiv Machine Learning

Jul 7

Incentivizing Vision Language Models to Search for Long Video Question Answering

arXiv:2607. 02959v1 Announce Type: cross Abstract: We introduce VSeek, an agentic framework that transforms long-video question answering (LVQA) from a passive, single-pass perception task into a multi-turn retrieval process.

By Harsh Goel, S P Sharan, Sahil Shah, Minkyu Choi, Joungbin An, Kristen Grauman, Sandeep P. Chinchali

llms agents nlp reinforcement-learning benchmarks

Get the daily briefing