Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

20,687 stories · RSS feed

arXiv Machine Learning

Jun 19

Where Does Social Reasoning Come From? Capability Provenance in Language Models

arXiv:2606. 19625v1 Announce Type: cross Abstract: We use training-data attribution as an interpretable tool for capability discovery, mapping which regions of the pretraining corpus support social-reasoning versus STEM-reasoning in OLMo3-7B.

By Glenn Matlin, Chandreyi Chakraborty, Saehee Eom, Mika Okamoto, Rayan Castilla, Louis Jaburi, Alvin Deng, Taywon Min, Lucia Quirke, Stella Biderman, Mark Riedl

llms benchmarks

arXiv AI

Jun 19

Simulation of Language Evolution under Regulated Social Media Platforms: A Synergistic Approach of Large Language Models and Genetic Algorithms

arXiv:2502. 19193v2 Announce Type: replace-cross Abstract: Social media platforms frequently impose restrictive policies to moderate user content, prompting the emergence of creative evasion language strategies.

By Jinyu Cai, Yusei Ishimizu, Mingyue Zhang, Munan Li, Jialong Li, Kenji Tei

llms agents benchmarks

arXiv AI

Jun 19

DeFrame: Debiasing Large Language Models Against Framing Effects

arXiv:2602. 04306v2 Announce Type: replace-cross Abstract: As large language models (LLMs) are increasingly deployed in real-world applications, ensuring their fair responses across demographics has become crucial.

By Kahee Lim, Soyeon Kim, Steven Euijong Whang

llms benchmarks safety

arXiv AI

Jun 19

A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models

arXiv:2606. 19868v1 Announce Type: new Abstract: Although large language models (LLMs) have shown strong capabilities across a wide range of tasks, their outputs often remain unreliable and may contain hallucinations, making uncertainty estimation (UE) essential for building trustworthy LLMs.

By Jiayi Wang, Xu-Yao Zhang

llms agents benchmarks safety

arXiv AI

Jun 19

Benchmarking Agentic Review Systems

arXiv:2606. 19749v1 Announce Type: new Abstract: A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated.

By Dang Nguyen, Wanqing Hao, Yanai Elazar, Chenhao Tan

llms agents benchmarks

arXiv AI

Jun 19

Learning to Prompt: Improving Student Engagement with Adaptive LLM-based High-School Tutoring

arXiv:2606. 20138v1 Announce Type: new Abstract: LLMs can personalize education, although current static-prompt tutoring systems struggle to adapt to diverse academic disciplines.

By Po-Chin Chang, Nicholas Hogan, Aske Plaat, Michiel T. van der Meer

llms robotics benchmarks

arXiv AI

Jun 19

DF3DV-1K: A Large-Scale Dataset and Benchmark for Distractor-Free Novel View Synthesis

arXiv:2604. 13416v2 Announce Type: replace-cross Abstract: Advances in radiance fields have enabled photorealistic novel view synthesis.

By Cheng-You Lu, Yi-Shan Hung, Wei-Ling Chi, Hao-Ping Wang, Charlie Li-Ting Tsai, Yu-Cheng Chang, Yu-Lun Liu, Thomas Do, Chin-Teng Lin

diffusion fine-tuning benchmarks

arXiv AI

Jun 19

CRAX: Fast Safe Reinforcement Learning Benchmarking

arXiv:2606. 20376v1 Announce Type: cross Abstract: Safety is a core concern for deploying reinforcement learning (RL) agents in real-world domains such as robotics and autonomous driving.

By Tristan Tomilin, Mourad Boustani, Mickey Beurskens, Thiago D. Sim\~ao

agents reinforcement-learning robotics benchmarks safety

arXiv Machine Learning

Jun 19

EFIQA: Explainable Fundus Image Quality Assessment via Anatomical Priors

arXiv:2606. 20108v1 Announce Type: cross Abstract: Image quality control is vital for a wide range of downstream applications.

By Pengwei Wang, Jos\'e Morano, Qian Wan, Hrvoje Bogunovi\'c

benchmarks

arXiv AI

Jun 19

Detecting Hallucinations for Large Language Model-based Knowledge Graph Reasoning

arXiv:2606. 19351v1 Announce Type: cross Abstract: Knowledge graph (KG) reasoning infers new knowledge from existing facts and is widely applied in question answering, recommendation, and decision support.

By Xinyan Zhu, Yaoqi Liu, Yue Gao, Huadong Ma, Cheng Yang, Chuan Shi

llms nlp benchmarks safety

arXiv AI

Jun 19

Implicit Semantic-Aware Communication Based on Hypergraph Reasoning

arXiv:2606. 20162v1 Announce Type: new Abstract: Semantic-aware communication has emerged as a transformative paradigm for next-generation communication systems, shifting the fundamental goal from transmitting bit-level symbols to reliably recovering and understanding the semantic meaning of information.

By Yiwei Liao, Shurui Tu, Yong Xiao, Yingyu Li, Guangming Shi

rag benchmarks

arXiv Machine Learning

Jun 19

MassSpecGym in the Wild: Uncovering and Correcting Evaluation Pitfalls in AI-Driven Molecule Discovery

arXiv:2606. 19624v1 Announce Type: new Abstract: Reliable benchmarking is critical for developing machine learning models for tandem mass spectrometry (MS/MS) based molecule discovery.

By Hongxuan Liu, Roman Bushuiev, Ivy Lightheart, Mrunali Manjrekar, Anton Bushuiev, Magdalena Lederbauer, Filip Jozefov, Yinkai Wang, Soha Hassoun, Josef Sivic, James Taylor, Runzhong Wang, David Healey, Tom\'a\v{s} Pluskal, Connor W. Coley

benchmarks

arXiv Machine Learning

Jun 19

Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation

arXiv:2510. 08807v2 Announce Type: replace-cross Abstract: From loco-motion to dextrous manipulation, humanoid robots have made remarkable strides in demonstrating complex full-body capabilities.

By Zhenyu Zhao, Hongyi Jing, Xiawei Liu, Jiageng Mao, Abha Jha, Hanwen Yang, Rong Xue, Sergey Zakharov, Vitor Guizilini, Yue Wang

agents robotics multimodal benchmarks

arXiv Machine Learning

Jun 19

On the Redundancy of Timestep Embeddings in Diffusion Models

arXiv:2606. 20416v1 Announce Type: new Abstract: Diffusion models rely heavily on explicit timestep embeddings to modulate the denoising process across various noise scales.

By Jos\'e A. Ch\'avez

llms rag diffusion benchmarks

arXiv AI

Jun 19

SL-S4Wave: Self-Supervised Learning of Physiological Waveforms with Structured State Space Models

arXiv:2606. 19888v1 Announce Type: cross Abstract: Modeling long-sequence medical time series data, such as electrocardiograms (ECG), poses significant challenges due to high sampling rates, multichannel signal complexity, inherent noise, and limited labeled data.

By Feng Wu, Harsh Deep, Eric Lehman, Sanyam Kapoor, Guoshuai Zhao, Rahul Krishnan, Gari Clifford, Li-wei H Lehman

benchmarks

arXiv Machine Learning

Jun 19

Shifting-based Optimizable Linear Relaxations for General Activation Functions

arXiv:2606. 20292v1 Announce Type: new Abstract: The use of neural networks (NNs) is rapidly increasing, including in safety- and security-critical domains.

By Philipp Kern, L\'aszl\'o Antal, Erika \'Abr\'aham, Carsten Sinz

benchmarks safety

arXiv Machine Learning

Jun 19

EQPO: Equitable Group Relative Policy Optimization for Clinical Reasoning

arXiv:2510. 19893v2 Announce Type: replace Abstract: Medical AI systems demonstrated impressive diagnostic performance, yet they routinely show uneven accuracy across demographic groups, disadvantaging underrepresented populations.

By Shiqi Dai, Wei Dai, Jiaee Cheong, Paul Pu Liang

reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 19

Scalable Training of Spatially Grounded 2D Vision-Language Models for Radiology

arXiv:2606. 20477v1 Announce Type: cross Abstract: We study how to train visually grounded vision-language models (VLMs) for radiology without manual spatial annotations.

By Yusuf Salcan (Computer Vision Group, University of Freiburg, Germany, CRIION-AI Lab, Freiburg, Germany), Simon Ging (Computer Vision Group, University of Freiburg, Germany, Adaptive & Agentic AI), Robin Schirrmeister (Department of Radiology, Medical Center -- University of Freiburg, Germany), Philipp Arnold (Department of Radiology, Medical Center -- University of Freiburg, Germany), Elmar Kotter (Department of Radiology, Medical Center -- University of Freiburg, Germany), Behzad Bozorgtabar (Adaptive & Agentic AI), Thomas Brox (Computer Vision Group, University of Freiburg, Germany)

llms computer-vision nlp fine-tuning multimodal benchmarks

arXiv Machine Learning

Jun 19

A Unified Perspective on the Dynamics of Deep Transformers

arXiv:2501. 18322v2 Announce Type: replace Abstract: Transformers, which are state-of-the-art in most machine learning tasks, represent the data as sequences of vectors called tokens.

By Val\'erie Castin, Pierre Ablin, Jos\'e Antonio Carrillo, Gabriel Peyr\'e

llms benchmarks

arXiv Machine Learning

Jun 19

Marginal Advantage Accumulation for Memory-Driven Agent Self-Evolution

arXiv:2606. 20475v1 Announce Type: new Abstract: In batch-style trace distillation, the same memory operation may receive contradictory feedback across different batches.

By Mingyu Yang, Keye Zheng, Congchao Cheng, Yujie Liu, Xingkang Lu, Fan Jiang, Yefei Zheng

agents efficiency benchmarks

Get the daily briefing