Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv AI

Jul 10

From Triggers to Emotions: A CPM-Grounded Appraisal Multi-Agent for Dynamic Emotional Evolution in Persona-Based Dialogue

arXiv:2607. 07824v1 Announce Type: cross Abstract: Large Language Models (LLMs) have substantially advanced persona-based dialogue agents for emotion-sensitive role simulation in healthcare, education, counseling, customer service, and interactive storytelling.

By Jingyao Cai, Shuaijun Liu, Abdul Rehman, Yutong Guo, Qin Tian, Thomas Dolby, Sue Green, Chantel Cox, Xiaosong Yang

llms agents benchmarks

arXiv AI

Jul 10

Adaptive Generation of Bias-Eliciting Questions for LLMs

arXiv:2510. 12857v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are now widely deployed in user-facing applications, reaching hundreds of millions of users worldwide.

By Robin Staab, Jasper Dekoninck, Maximilian Baader, Martin Vechev

llms benchmarks safety

arXiv AI

Jul 10

path_boost: A Python Package for Interpretable Graph-Level Prediction using Path-Based Gradient Boosting

arXiv:2607. 07935v1 Announce Type: cross Abstract: We present path_boost, a Python package for interpretable supervised learning on graph-structured input data.

By Claudio Meggio, Johan Pensar, Riccardo De Bin

benchmarks

arXiv AI

Jul 10

Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning

arXiv:2607. 07761v1 Announce Type: new Abstract: Large language models (LLMs) have emerged as important tools in healthcare, showing growing potential for clinical reasoning and patient care.

By Qi Peng, Jiatong Li, Sirui Huang, Yiyang Jiang, Kaisong Gong, Ronger Ding, Shijie Ye, Changmeng Zheng, Yi Cai, Xiaobo Yang, Jin Huang, Xiao-Yong Wei, Qing Li

llms benchmarks safety

arXiv AI

Jul 10

Echoes: A semantically-aligned music deepfake detection dataset

arXiv:2603. 23667v2 Announce Type: replace-cross Abstract: We introduce Echoes, a new dataset for music deepfake detection designed for training and benchmarking detectors under realistic and provider-diverse conditions.

By Octavian Pascu, Dan Oneata, Horia Cucu, Nicolas M. Muller

benchmarks safety

arXiv AI

Jul 10

CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

arXiv:2607. 08093v1 Announce Type: new Abstract: Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use.

By Andrej Leban, Yuekai Sun

llms agents benchmarks

arXiv AI

Jul 10

Omni-Sleep: A Sleep Foundation Model via Hierarchical Contrastive Learning of CNS--ANS Dynamic

arXiv:2607. 07720v1 Announce Type: cross Abstract: Sleep physiology arises from the coordinated dynamics of the central nervous system (CNS) and autonomic nervous system (ANS), as reflected by multimodal polysomnography signals including EEG, EOG, EMG, ECG, and respiration.

By Zhoujie Hou, Song Wang, Kexin Lou, Mo Wang, Chen Wei, Quanying Liu

multimodal benchmarks

arXiv AI

Jul 10

PS4: Proxy-Supervised Joint Training for Real Target Speaker Extraction

arXiv:2607. 08111v1 Announce Type: cross Abstract: Training target speaker extraction (TSE) models for real conversational mixtures remains challenging because large-scale training corpora and clean target speech for supervision are unavailable.

By Wanyi Ning, Wei Zhou, Yingpeng Li, Yinshang Guo, Haitao Qian, Yiming Cheng

fine-tuning multimodal benchmarks

arXiv AI

Jul 10

The Phasor Transformer: Resolving Attention Bottlenecks on the Unit Circle

arXiv:2603. 17433v2 Announce Type: replace-cross Abstract: Transformer models have redefined sequence learning, yet dot-product self-attention introduces a quadratic token-mixing bottleneck for long-context time-series.

By Dibakar Sigdel

llms benchmarks

arXiv Machine Learning

Jul 10

When Does Continual Learning Require Learning

arXiv:2607. 07847v1 Announce Type: new Abstract: As large language models (LLMs) become increasingly capable, the next question is how can we enable models to continually learn?

By Anne Harrington, Nayan Saxena, Michael Murphy, Anastasia Borovykh, Zeyu Yun, Sridhar Kamath, Ara Eindra Kyi, Trevor Darrell, Jitendra Malik, Yutong Bai

llms agents reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jul 10

SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Scheduling

arXiv:2607. 08565v1 Announce Type: cross Abstract: LLM scheduling is critical to serving, yet it remains unclear how well existing designs fit agentic serving--with LLM requests issued by agents instead of humans.

By Jiahao Wang, Kaizhan Lin, Kaixi Zhang, Jinbo Han, Xingda Wei, Sijie Shen, Chenguang Fang, Wenyuan Yu, Rong Chen, Haibo Chen

llms agents benchmarks

arXiv AI

Jul 10

DeepTutor: Towards Agentic Personalized Tutoring

arXiv:2604. 26962v3 Announce Type: replace-cross Abstract: Education is one of the most promising real-world applications for Large Language Models (LLMs).

By Bingxi Zhao, Jiahao Zhang, Xubin Ren, Zirui Guo, Tianzhe Chu, Yi Ma, Chao Huang

llms rag agents benchmarks safety

arXiv AI

Jul 10

MentalHospital: A Virtual Environment for Evaluating Psychiatric Clinical Encounters

arXiv:2607. 08257v1 Announce Type: new Abstract: Large language models (LLMs) have shown strong performance on isolated psychiatric tasks, including dialogue, diagnosis, and treatment planning, yet existing benchmarks rarely simulate complete psychiatric clinical encounters.

By Yuming Yang, Xiao Sun, Yuanwei Zou, Zhengxiao Wu, Yun Chen, Jiang Zhong, Haoyang Zeng, Jingwang Huang, Kaiwen Wei

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jul 10

Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems

arXiv:2607. 08013v1 Announce Type: new Abstract: Federated Learning (FL) empowers multiple clients to collaboratively learn a model, enlarging the training data of each client for high accuracy while protecting data privacy.

By Shuo Huai, Di Liu, Hao Kong, Xiangzhong Luo, Weichen Liu, Ravi Subramaniam, Christian Makaya, Qian Lin

benchmarks safety

arXiv Machine Learning

Jul 10

DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks

arXiv:2607. 07946v1 Announce Type: cross Abstract: DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.

By Wenqi Huang, Charley Lee, Leonard Tng, Serena Ge

llms agents benchmarks

arXiv AI

Jul 10

LSRM: High-Fidelity Object-Centric Reconstruction via Scaled Context Windows

arXiv:2604. 05182v2 Announce Type: replace-cross Abstract: We introduce the Large Sparse Reconstruction Model to study how scaling transformer context windows affects feed-forward 3D reconstruction.

By Zhengqin Li, Cheng Zhang, Jakob Engel, Zhao Dong

llms benchmarks

arXiv Machine Learning

Jul 10

Scalable and Trustworthy Earth Observation Foundation Models

arXiv:2607. 07758v1 Announce Type: new Abstract: Foundation models (FMs) have transformed machine learning from isolated task-specific model development toward general-purpose models pretrained on broad data and adapted to multiple downstream tasks.

By Syed Usama Imtiaz, Mitra Nasr Azadani, Nasrin Alamdari

reinforcement-learning multimodal benchmarks

arXiv AI

Jul 10

Bridging Cognitive Neuroscience and Graph Intelligence: Hippocampus-Inspired Multi-View Hypergraph Learning for Web Finance Fraud

arXiv:2601. 11073v3 Announce Type: replace-cross Abstract: Online financial services constitute an essential component of contemporary web ecosystems, yet their openness introduces substantial exposure to fraud that harms vulnerable users and weakens trust in digital finance.

By Rongkun Cui, Nana Zhang, Kun Zhu, Qi Zhang

benchmarks safety

arXiv Machine Learning

Jul 10

Frequency-Domain Multi-Modality Transportation Modeling

arXiv:2607. 08475v1 Announce Type: new Abstract: Multi-modality transportation refers to urban systems composed of multiple transportation modes, such as traffic flow and public transit, whose dynamics are coupled by shared temporal patterns.

By Jiewen Deng, Hangchen Liu, Junchen Li, Boyuan Zhang, Renhe Jiang

benchmarks

arXiv Machine Learning

Jul 10

Ensemble Diversity Optimization for Subjective Supervision

arXiv:2607. 08493v1 Announce Type: new Abstract: Subjective NLP tasks often exhibit systematic annotator disagreement, requiring models that represent uncertainty rather than collapse it.

By Xia Cui, Ziyi Huang, N. R. Abeynayake

nlp benchmarks safety

Get the daily briefing