Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,880 stories · RSS feed

arXiv Machine Learning

Jun 10

Rare Event Analysis via Stochastic Optimal Control

arXiv:2604. 13213v2 Announce Type: replace-cross Abstract: Rare events such as conformational changes in biomolecules, phase transitions, and chemical reactions are central to the behavior of many physical systems, yet they are extremely difficult to study computationally because unbiased simulations seldom produce them.

By Yuanqi Du, Jiajun He, Dinghuai Zhang, Eric Vanden-Eijnden, Carles Domingo-Enrich

benchmarks

arXiv AI

Jun 10

CleanPatrick: A Benchmark for Image Data Cleaning

arXiv:2505. 11034v2 Announce Type: replace-cross Abstract: Robust machine learning depends on clean data, yet current image data cleaning benchmarks rely on synthetic noise or narrow human studies, limiting comparison and real-world relevance.

By Fabian Gr\"oger, Simone Lionetti, Philippe Gottfrois, Alvaro Gonzalez-Jimenez, Ludovic Amruthalingam, Elisabeth Victoria Goessinger, Hanna Lindemann, Marie Bargiela, Marie Hofbauer, Omar Badri, Philipp Tschandl, Arash Koochek, Matthew Groh, Alexander A. Navarini, Marc Pouly

benchmarks

arXiv AI

Jun 10

Whisfusion: Parallel ASR Decoding with Masked Diffusion

arXiv:2508. 07048v2 Announce Type: replace-cross Abstract: Autoregressive (AR) encoder-decoder models dominate high-quality multilingual ASR, but their left-to-right decoders make inference latency scale with transcript length.

By Taeyoun Kwon, Junhyuk Ahn, Taegeun Yun, Heeju Jwa, Yoonchae Choi, Siwon Park, Jongchan Kim, Hyungon Ryu, Hyuk-Jae Lee, Nam-Joon Kim

llms rag diffusion efficiency multimodal benchmarks

arXiv AI

Jun 10

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

arXiv:2606. 09864v1 Announce Type: cross Abstract: Key-value (KV) cache quantization is widely used to reduce Large Language Model (LLM) inference memory, yet existing evaluations solely focus on measuring perplexity and accuracy without assessing the safety impact.

By Bruce Changlong Xu, Adarsh Kumarappan, Mu Zhou

llms efficiency benchmarks safety

arXiv AI

Jun 10

Atomic Intent Reasoning: Bringing LLM Semantics to Industrial Cross-Domain Recommendations

arXiv:2606. 10357v1 Announce Type: cross Abstract: Cross-domain recommendation is a core problem in content-to-e-commerce platforms.

By Zhuohang Jiang, Yuxin Chen, Shijie Wang, Haohao Qu, Zhou Jindong, Wenqi Fan, Li Qing, Dongxu Liang, Jun Wang

llms efficiency benchmarks

arXiv AI

Jun 10

Content-Induced Spatial-Spectral Aggregation Network for Change Detection in Remote Sensing Images

arXiv:2606. 10328v1 Announce Type: cross Abstract: The integration of spatial and spectral information is beneficial to the improvement of change detection performance.

By Yunlong Liu, Zekai Zhang

benchmarks

arXiv AI

Jun 10

$\tau$-Rec: A Verifiable Benchmark for Agentic Recommender Systems

arXiv:2606. 10156v1 Announce Type: cross Abstract: As recommender systems transition toward agentic, multi-turn conversational interfaces, evaluation paradigms have struggled to keep pace.

By Bharath Sivaram Narasimhan, Karthik R Narasimhan

llms agents benchmarks

arXiv AI

Jun 10

When Attribution Patching Lies: Diagnosis and a Second-Order Correction

arXiv:2606. 09899v1 Announce Type: cross Abstract: A central goal of mechanistic interpretability is to identify which internal components causally drive a language model's behavior.

By Luyang Zhang, Jialu Wang

llms benchmarks safety

arXiv Machine Learning

Jun 10

Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data

arXiv:2606. 10789v1 Announce Type: new Abstract: Zero-shot learning (ZSL) for inertial measurement unit (IMU)-based human activity recognition (HAR) faces a central challenge: bridging the gap between sensor embeddings and semantic class representations.

By Anik Ghosh

llms rag benchmarks safety

arXiv Machine Learning

Jun 10

Informed Asymmetric Actor-Critic: Leveraging Privileged Signals Beyond Full-State Access

arXiv:2509. 26000v3 Announce Type: replace Abstract: Asymmetric reinforcement learning leverages privileged information available during training to improve learning under partial observability.

By Daniel Ebi, Damien Ernst, Klemens B\"ohm, Gaspard Lambrechts

reinforcement-learning benchmarks

arXiv AI

Jun 10

HiGR: Industrial-Scale Hierarchical Generative Slate Recommendation Framework in Tencent

arXiv:2512. 24787v3 Announce Type: replace-cross Abstract: Slate recommendation, which presents users with a ranked item list in a single display, is ubiquitous across mainstream online platforms.

By Yunsheng Pang, Zijian Liu, Yudong Li, Shaojie Zhu, Zijian Luo, Chenyun Yu, Sikai Wu, Shichen Shen, Cong Xu, Bin Wang, Kai Jiang, Chengxiang Zhuo, Zang Li

rag efficiency benchmarks safety

arXiv AI

Jun 10

Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts

arXiv:2606. 10334v1 Announce Type: new Abstract: Code-generating large language models (LLMs) increasingly produce visual artifacts such as charts, web pages, and slides by writing programs that are executed by non-differentiable renderers, committing to code before observing the render.

By Haoyu Dong

llms efficiency benchmarks safety

arXiv AI

Jun 10

Role-Agent: Bootstrapping LLM Agents via Dual-Role Evolution

arXiv:2606. 10917v1 Announce Type: new Abstract: Although Large Language Model (LLM) agents have demonstrated strong performance on complex tasks, their learning is often limited by inefficient interaction feedback and static training environments, which hinder broader generalization.

By Xucong Wang, Ziyu Ma, Shidong Yang, Tongwen Huang, Pengkun Wang, Yong Wang, Xiangxiang Chu

llms agents benchmarks safety

arXiv Machine Learning

Jun 10

Generative Archetype-Grounded Item Representations for Sequential Recommendation

arXiv:2606. 11023v1 Announce Type: cross Abstract: Sequential recommendation aims to predict users' next interaction with items by analyzing their historical behavior.

By Yifan Li, Jiahong Liu, Xinni Zhang, Hao Chen, Yankai Chen, Wenhao Yu, Jianting Chen, Irwin King

llms rag benchmarks

arXiv AI

Jun 10

Trace2Policy: From Expert Behavior Traces to Self-Evolving Decision Agents

arXiv:2606. 10457v1 Announce Type: new Abstract: Decision rules that enterprise experts apply tacitly -- in auditing, compliance, and contract review -- can be systematically recovered and improved through iterative error analysis.

By Junli Zha, Jinbo Wang, Chao Zhou, Xiang Song

llms agents efficiency benchmarks

arXiv AI

Jun 10

Using Probabilistic Programs to Train Inductive Reasoning in Large Language Models

arXiv:2606. 09856v1 Announce Type: cross Abstract: Post-training Large Language Models (LLMs) for reasoning typically focuses on deductive tasks such as mathematics and coding where correctness is verifiable.

By Liyi Zhang, Akshay K. Jagadish, Brenden M. Lake, Thomas L. Griffiths

llms agents fine-tuning benchmarks safety

arXiv AI

Jun 10

Bittensor Agent Arenas as a Trajectory Primitive: Distilling a Shopping Agent from ShoppingBench Subnet Traces

arXiv:2606. 10064v1 Announce Type: cross Abstract: Small-model agentic post-training is bottlenecked less by the algorithm than by the trajectory substrate it consumes.

By Shardul Bansal, Seth Schilbe, Jarrod Barnes

llms agents reinforcement-learning fine-tuning multimodal benchmarks

arXiv AI

Jun 10

READER: Robust Evidence-based Authorship Decoding via Extracted Representations

arXiv:2606. 10794v1 Announce Type: new Abstract: As agentic applications increasingly route user tasks through official and third-party LLM APIs, provenance becomes an operational question: which model generated a given black-box response?

By Jiaxu Liu, Sunnan Mu, Dong Huang, Liuyin Wang, Jing Shao, Jie Zhang

llms agents benchmarks

arXiv Machine Learning

Jun 10

Do Transformers Actually Help Intrusion Detection? A Temporal Sequence Evaluation on CIC-IDS2017

arXiv:2606. 11098v1 Announce Type: cross Abstract: Recent deep learning approaches for network intrusion detection increasingly incorporate temporal architectures such as recurrent networks and Transformers, often reporting near-perfect performance on CIC-IDS2017.

By Zach Moczkodan (Royal Military College of Canada, Kingston, Canada), Hany Ragab (Royal Military College of Canada, Kingston, Canada)

llms benchmarks

arXiv Machine Learning

Jun 10

When Design Rules Break: Benchmark Composition Determines Whether Label Informativeness Predicts GNN Aggregator Choice

arXiv:2606. 10249v1 Announce Type: new Abstract: We examine whether graph neural network (GNN) design rules generalize across benchmark families by studying aggregator selection (sum, mean, max) on 24 node-classification datasets spanning citation, heterophilic, LINKX Facebook-100, co-purchase, and co-authorship graphs.

By Neha Sharma, Ritesh Sharma

benchmarks

Get the daily briefing