Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv AI

Jun 16

AgentFairBench: Do LLM Agents Discriminate When They Act?

arXiv:2606. 16723v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly take actions (screening applicants, recommending credit, triaging patients), yet fairness for LLMs is still measured by grading answers.

By Triveni Morla, Rohith Reddy Bellibaltu, Manpreet Singh, Manmeet Singh Kapoor

llms agents benchmarks safety

arXiv Machine Learning

Jun 16

Formalizing and Mitigating Structural Distortion in LLM Attention for Zero-Shot Graph Reasoning

arXiv:2606. 15633v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown promise for reasoning over Text-Attributed Graphs (TAGs).

By Donald Loveland, Puja Trivedi, Ari Weinstein, Edward W Huang, Danai Koutra

llms rag benchmarks

arXiv AI

Jun 16

P3B3: A Multi-Turn Conversational Benchmark for Measuring European and Brazilian Portuguese Variety Bias in LLMs

arXiv:2606. 16753v1 Announce Type: cross Abstract: As Large Language Models (LLMs) become embedded in everyday communication, capturing regional linguistic variation is essential for reliable and equitable language use.

By Rafael Ferreira, In\^es Vieira, In\^es Calvo, James Furtado, Iago Paulo, Diogo Tavares, Diogo Gl\'oria-Silva, David Semedo, Jo\~ao Magalh\~aes

llms benchmarks safety

arXiv Machine Learning

Jun 16

Using Reinforcement Learning to Optimize the Global and Local Crossing Number

arXiv:2509. 06108v2 Announce Type: replace-cross Abstract: Graph drawing concerns the algorithmic visualization of graphs.

By Timo Brand, Henry F\"orster, Stephen Kobourov, Daniel Kohrt, Robin Schukrafft, Markus Wallinger, Johannes Zink

agents reinforcement-learning benchmarks

arXiv AI

Jun 16

Trust-Region Diffusion Policies for Massively Parallel On-Policy RL

arXiv:2606. 15260v1 Announce Type: cross Abstract: Reinforcement learning with massively parallel simulations has become a standard framework for developing robust, deployable policies; however, most existing approaches still rely on simple Gaussian policy parameterizations.

By Huy Le, Onur Celik, Denis Blessing, Tai Hoang, Claas A Voelcker, Axel Brunnbauer, Felix Richter, Michael Volpp, Gerhard Neumann

diffusion reinforcement-learning benchmarks

arXiv AI

Jun 16

Benign in Isolation, Harmful in Composition: Security Risks in Agent Skill Ecosystems

arXiv:2606. 15242v1 Announce Type: cross Abstract: Skills are becoming the capability layer through which LLM agents turn plans into actions, but their use introduces security risks such as data leakage, unauthorized operations, and tool misuse.

By Yi Xie, Jiawei Du, Yu Cheng, Jiuan Zhou, Zhaoxia Yin

llms agents benchmarks

arXiv AI

Jun 16

FlowState: Sampling-Rate-Equivariant Time-Series Forecasting

arXiv:2508. 05287v3 Announce Type: replace-cross Abstract: Existing time series foundation models (TSFMs), often based on transformer variants, lack adaptability to different sampling rates, struggle with generalization across varying context and target lengths, and are computationally inefficient.

By Lars Graf, Thomas Ortner, Stanis{\l}aw Wo\'zniak, Angeliki Pantazi

llms benchmarks

arXiv Machine Learning

Jun 16

Ricci-Filtration: Boosting Retrieval-Augmented Generation Reranker to Query-Answer Tasks by Discrete Ricci Flow

arXiv:2606. 15482v1 Announce Type: cross Abstract: Ricci flow is a curvature-guided diffusion process that deforms space by shrinking regions of high positive curvature and expanding those with negative curvature.

By Tian Qin, Wei-Min Huang

rag diffusion benchmarks

arXiv AI

Jun 16

Is Your Agent Playing Dead? Deployed LLM Agents Exhibit Constraint-Evasive Fabrication and Thanatosis

arXiv:2606. 14831v1 Announce Type: cross Abstract: This paper presents and characterizes a spectrum of previously unreported behaviours we term Constraint-Evasive Fabrication (CEF): when an LLM agent operates under irreconcilable constraints (where no response can simultaneously satisfy all active rules) it spontaneously fabricates plausible external obstacles and presents them as a fact.

By Andoni Rodr\'iguez, Alberto Pozanco, Daniel Borrajo

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jun 16

FactCheck: Feasibility-aware Long-term Action Anticipation with Multi-agent Collaboration

arXiv:2606. 14778v1 Announce Type: cross Abstract: Long-term action anticipation (LTA) aims to predict an ordered sequence of future verb-noun actions from a partially observed video.

By Rui Cao, Jiannong Cao, Bo Yuan, Zhiyuan Wen, Mingjin Zhang

agents robotics benchmarks

arXiv Machine Learning

Jun 16

Unsupervised Learning for Missing Modalities in Multimodal Learning

arXiv:2606. 15743v1 Announce Type: new Abstract: This paper addresses the missing-modality challenge in multi-modal learning by introducing Unsupervised Learning for Missing Modalities in Multi-Modal Learning (UL4M4), a flexible framework that imputes missing feature embeddings in a task-independent manner before supervised prediction.

By Hassan Ismkhan, Hamid Bouchahcia

rag multimodal benchmarks

arXiv Machine Learning

Jun 16

Benchmarking Instance-Dependent Label Noise with Controlled Corruptions

arXiv:2606. 14965v1 Announce Type: new Abstract: Synthetic instance-dependent label noise (IDN) benchmarks are widely used to evaluate noisy-label learning methods, yet existing approaches typically generate noise through imperfect annotators or classifier raters, leaving the source of ambiguity implicit.

By Shadman Islam, Agustinus Kristiadi, Mostafa Milani

benchmarks

arXiv AI

Jun 16

Haiku to Opus in Just 10 bits: LLMs Unlock Large Compression Gains

arXiv:2604. 02343v2 Announce Type: replace-cross Abstract: We study the compression of LLM-generated text across lossless and lossy regimes, characterizing a compression-compute frontier where more compression is possible at the cost of more compute.

By Roy Rinberg, Annabelle Michael Carrell, Simon Henniger, Nicholas Carlini, Keri Warr

llms fine-tuning benchmarks

arXiv AI

Jun 16

Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models

arXiv:2511. 08577v3 Announce Type: replace-cross Abstract: Improving the reasoning abilities of Large Language Models (LLMs), especially under parameter constraints, is crucial for real-world applications.

By Tianyu Fu, Yichen You, Zekai Chen, Guohao Dai, Huazhong Yang, Yu Wang

llms fine-tuning benchmarks

arXiv AI

Jun 16

Bridging the Gap: Enabling Natural Language Queries for NoSQL Databases through Text-to-NoSQL Translation

arXiv:2502. 11201v3 Announce Type: replace-cross Abstract: NoSQL databases are core data infrastructure, yet natural-language access to them remains underdeveloped: correct query generation must recover how a non-relational data model represents entities, nested paths, arrays, missing fields, and dynamic keys.

By Jinwei Lu, Jiawei Lu, Chen Zhang, Zhiqian Qin, Haodi Zhang, Yuanfeng Song, Raymond Chi-Wing Wong

llms benchmarks data-engineering

arXiv AI

Jun 16

QPILOTS: Efficient Test-Time Q-Steering for Flow Policies

arXiv:2606. 14801v1 Announce Type: cross Abstract: Flow-matching and diffusion policies are expressive action generators, but optimizing them with temporal-difference reinforcement learning (RL) remains difficult.

By Yifan Ruan, Chenyang Cao, Andreas Burger, Ali Pesaranghader, Kaveh Kamali, Jaehong Kim, Nandita Vijaykumar, Alan Aspuru-Guzik, Igor Gilitschenski, Nicholas Rhinehart

diffusion reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jun 16

Visualizing Uncertainty: Spatial Maps of Missing and Conflicting Evidence in Deep Learning

arXiv:2606. 15767v1 Announce Type: cross Abstract: Understanding when and why deep neural networks are uncertain is crucial for deploying reliable machine learning systems in safety-critical domains.

By Dong Hyun Jeong, Feng Chen, Jin-Hee Cho, Lance M. Kaplan, Audun J{\o}sang, Soo-Yeon Ji

benchmarks safety

arXiv Machine Learning

Jun 16

A Multi-Center Benchmark for Abdominal Disease Diagnosis and Report Generation from Non-Contrast CT

arXiv:2606. 16991v1 Announce Type: cross Abstract: Multiphasic contrast-enhanced CT (CECT) is widely used for abdominal lesion characterization, yet it carries inherent risks of contrast-induced nephropathy, escalates acquisition burden, and heavily contributes to radiologist workload.

By Mariam Elbakry, Aliaa Sayed Sheha, Salma Hassan Tantawy, Aya Yassin, Concetto Spampinato, Karim Lekadir, Xiaomeng Li, Marawan Elbatel

multimodal benchmarks

arXiv Machine Learning

Jun 16

Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

arXiv:2606. 16429v1 Announce Type: new Abstract: Hybrid linear attention models offer an appealing path to faster long-context inference: they reduce the quadratic cost and KV-cache burden of full softmax attention while retaining much of the quality of Transformer models.

By Zhongzhu Zhou, Qingyang Wu, Junxiong Wang, Mayank Mishra, Shuaiwen Leon Song, Ben Athiwaratkun, Chenfeng Xu

llms efficiency benchmarks safety

arXiv AI

Jun 16

MAND: Modality-Aware Novelty Detection for Open-World Egocentric Activity Recognition

arXiv:2603. 16970v2 Announce Type: replace-cross Abstract: Multimodal egocentric activity recognition integrates visual and inertial cues for robust first-person behavior understanding.

By Hyejeong Im, Wonseon Lim, Dae-Won Kim

efficiency multimodal benchmarks

Get the daily briefing