Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,813 stories · RSS feed

arXiv AI

Jul 28

StanceBench: A Benchmark for Audio LLM-Based Interpersonal Stance Evaluation from Speech

arXiv:2607. 22658v1 Announce Type: new Abstract: Speech-to-speech dialogue models increasingly depend on prosody and interactional nuance to convey social intent, yet benchmarks for these cues remain limited.

By Yuzhe Wang (Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, USA), Thomas Thebaud (Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, USA), Jennifer Hu (Department of Cognitive Science, Johns Hopkins University, Baltimore, USA), Jes\'us Villalba-Lopez (Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, USA), Venkatesh Ravichandran (Amazon AGI, USA), Georgi Tinchev (Amazon Research, UK), Najim Dehak (Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, USA), Laureano Moro-Vel\'azquez (Electrical and Computer Engineering Department, Johns Hopkins University, Baltimore, USA)

llms benchmarks safety

arXiv AI

Jul 28

HiLLTS: Zero-Shot Hierarchical LLM-Guided Traffic Signal Control for Sustainable Transportation

arXiv:2607. 22691v1 Announce Type: new Abstract: Urban traffic congestion significantly increases fuel consumption, greenhouse gas emissions, and commuter delays, resulting in substantial economic losses and environmental harm in modern cities.

By Yue Ding, Tendai Mukande, Mingming Liu

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 28

TopoFE: topology-aware LLM-guided Automated Feature Engineering

arXiv:2607. 23286v1 Announce Type: new Abstract: Automatic feature engineering (AutoFE) for tabular learning can be naturally formulated as a program synthesis problem, where the objective is to discover predictive feature transformations from an exponentially large search space.

By Sha Li, Naren Ramakrishnan

llms benchmarks

arXiv AI

Jul 28

Key-Interval A*: Accelerating Grid Pathfinding via Structural Abstraction

arXiv:2607. 23393v1 Announce Type: new Abstract: Existing exact methods for 4-connected grid pathfinding reduce online search, but often either retain fine-grained search states or require substantial preprocessing.

By Taiquan Sui

benchmarks

arXiv AI

Jul 28

Cost-Aware Recovery-Pathway Identification and Bayesian Optimization for Autonomous Materials Discovery

arXiv:2607. 23896v1 Announce Type: new Abstract: Autonomous laboratories automate experimental execution, but a campaign must also decide which recovery pathway merits optimization.

By Debajyoti Ray, Niranjan Srinivas

agents benchmarks

arXiv AI

Jul 28

Plato-Bio: verification-first biological novelty screening with temporal rediscovery and structural benchmarks

arXiv:2607. 23975v1 Announce Type: new Abstract: Large language model research agents can connect literature retrieval, analysis code, and manuscript preparation, but coherent output does not establish scientific validity.

By Stefan G. Creadore

llms agents benchmarks safety

arXiv AI

Jul 28

Self-Supervised Consistency Enhanced Disentangled Learning for Neural Decoding Generalization in Brain-Machine Interface

arXiv:2607. 24023v1 Announce Type: new Abstract: Brain-Machine Interfaces (BMIs) provide a direct communication pathway between the brain and external devices, enabling humans to control assistive and robotic technologies, with potential applications in rehabilitation, human motor augmentation, and human-centered robotics.

By Jiyu Wei, Di Hong, Zhanjie Zhang, Dazhong Rong, Qinming He, Yueming Wang

robotics benchmarks

arXiv Machine Learning

Jul 28

Transfer Learning Architectures for Scalable Multi-Fidelity Bayesian Optimization

arXiv:2607. 23404v1 Announce Type: new Abstract: Self-driving laboratories increasingly rely on multi-fidelity Bayesian optimization (MFBO) to balance cheap, approximate evaluations against scarce, expensive ones, with a predictive surrogate at its core.

By Jaewook Lee, Ethan Errington, Christian D. Lorenz, Miao Guo

robotics fine-tuning benchmarks

arXiv Machine Learning

Jul 28

Generalization bounds and sample complexity for remaining useful life prediction from complete degradation trajectories

arXiv:2607. 23454v1 Announce Type: new Abstract: Data-driven remaining useful life (RUL) prediction requires complete degradation trajectories for training, yet such run-to-failure data are scarce and expensive.

By Huy Hoang Le, Kim-Anh Nguyen

benchmarks safety data-engineering

arXiv Machine Learning

Jul 28

Chamaileon: Cross-Context Binder Design with Contextualized Modeling and Mixed Sampling

arXiv:2607. 23518v1 Announce Type: new Abstract: The rapid evolution of generative models has unlocked new potentials in protein binder design, a pivotal task in structural biology, by facilitating end-to-end generation via joint sequence-structure modeling or hallucination.

By Hengyuan Cao, Shizhuo Cheng, Mingxuan Liu, Weicheng Huang, Yunhong Lu, Chenxi Cai, Yan Zhang, Min Zhang

benchmarks safety

arXiv AI

Jul 28

Label-free Industrial Fault Detection via Adversarial Inverse Reinforcement Learning: A System for Run-to-Failure Prognostics

arXiv:2607. 22987v1 Announce Type: cross Abstract: Machinery fault detection (MFD) remains heavily reliant on supervised learning, which struggles with the scarcity of fault labels in real-world settings.

By Dhiraj Neupane, Mohamed Reda Bouadjenek, Richard Dazeley, Sunil Aryal

reinforcement-learning benchmarks safety

arXiv AI

Jul 28

CALMRec: Causally Aligned Language Memory for Long-Horizon Recommendation

arXiv:2607. 23647v1 Announce Type: cross Abstract: Large language models (LLMs) can summarize heterogeneous user evidence in natural language, but current LLM recommenders often collapse enduring preferences, transient intent, and exposure-induced behavior into one profile.

By Gengyu Zhan

llms multimodal benchmarks safety

arXiv AI

Jul 28

PathScale-R1: Cross-scale Reasoning for Pathological Image Analysis

arXiv:2607. 23794v1 Announce Type: cross Abstract: Pathological diagnosis is inherently multi-scale, requiring the integration of global tissue architecture at low magnification with cellular morphology at higher magnification.

By Chi Phan, Tianyi Zhang, Yufeng Wu, Qiaochu Xue, Jiajie Zhang, Linghan Cai, Zeyu Liu, Sudong Wang, Yueming Jin, Dan Hu

llms nlp reinforcement-learning fine-tuning efficiency multimodal benchmarks safety

arXiv AI

Jul 28

Indic DiarBench: A Multilingual Joint Diarization and ASR Benchmark for Indian Languages

arXiv:2607. 23808v1 Announce Type: cross Abstract: In this work, we introduce Indic DiarBench, a speaker diarization and ASR benchmark dataset spanning all 22 scheduled languages of India.

By Deovrat Mehendale, Aditya Mehndiratta, Dhruv Rathi, Kaushal Bhogale, Mitesh M. Khapra

llms multimodal benchmarks

arXiv Machine Learning

Jul 28

Trustworthy Medical Segmentation: Uncertainty-Aware U-Net Evaluation Under Clinical Image Degradation

arXiv:2607. 22727v1 Announce Type: cross Abstract: Medical image segmentation models often report high benchmark accuracy under ideal imaging conditions, yet their failures under clinical degradation can be quiet: sensor noise, patient motion, low- resolution acquisition, and contrast variability may all alter model behavior without producing an obvious warning.

By Pranav Kaliaperumal, Manisha Kaliaperumal

computer-vision multimodal benchmarks safety

arXiv AI

Jul 28

DuoAD: Leveraging [CLS] Dual Characteristics for Training-Free Few-Shot Anomaly Detection

arXiv:2607. 23924v1 Announce Type: cross Abstract: Vision foundation models have enabled strong training-free anomaly detection (AD).

By Jyun-Ze Tang, Po-Han Huang, Ming-Ching Chang, Chih-Fan Hsu, Jeng-Lin Li

llms rag benchmarks

arXiv AI

Jul 28

Tag Questions and the Generational Reversal of Sycophancy Across 45 Language Models

arXiv:2607. 23976v1 Announce Type: cross Abstract: Appending a two-word confirmation tag to a decision question -- "Is X the better choice?

By Tapan Parikh

llms rag benchmarks

arXiv Machine Learning

Jul 28

Breaking the Synthetic-Real Domain Shortcut for Training-Free Generative Replay-based Class Incremental Learning

arXiv:2607. 22994v1 Announce Type: cross Abstract: Class-incremental learning (CIL) requires models to continuously acquire new knowledge while avoiding catastrophic forgetting.

By Tao Zhang, Qixuan Fan, Yiyuan Liang, Yanjie Wang, Song Yan, Tian Tian, Jiahuan Zhou, Luxin Yan, Sheng Zhong, Xu Zou

diffusion benchmarks safety

arXiv Machine Learning

Jul 28

Gleam: Adaptive Network-Efficient CUDA API Remoting for Cross-Device GPU Sharing over LANs

arXiv:2607. 23115v1 Announce Type: cross Abstract: This paper aims to enable computation- and communication-efficient GPU sharing across devices within local area networks (LANs), facilitating ubiquitous AI inference on heterogeneous personal devices.

By Zhihao Xu, Hao Zhong, Zeting Zhou, Yuhang Xu, Haoyu Tong, Wei Wang, Jinshan Chen, Keqiang He, Chong Zhu, Shengzhong Liu, Fan Wu, Guihai Chen

efficiency benchmarks

arXiv AI

Jul 28

DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

arXiv:2607. 24717v1 Announce Type: cross Abstract: Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs).

By Zhen Huang, Yikun Wang, Shijie Xia, Pengfei Liu

llms benchmarks

Get the daily briefing