Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv Machine Learning

Jun 17

The Critical Role of Model Selection in Causal Inference: A Comparative Analysis of Classification Models within the InferBERT Framework for Pharmacovigilance

arXiv:2606. 17113v1 Announce Type: new Abstract: Distinguishing causal adverse drug events (ADEs) from spurious correlations remains a central challenge in pharmacovigilance.

By Csaba Kiss, Roland Molontay, Gabriele Pergola

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 17

Searching Neural Architectures for Sensor Nodes on IoT Gateways

arXiv:2505. 23939v2 Announce Type: replace Abstract: This paper presents an automatic method for the design of Neural Networks (NNs) at the edge, enabling Machine Learning (ML) access even in privacy-sensitive Internet of Things (IoT) applications.

By Andrea Mattia Garavagno, Edoardo Ragusa, Antonio Frisoli, Paolo Gastaldo

benchmarks safety

arXiv AI

Jun 17

Online LLM Selection via Constrained Bandits with Time-Varying Demand

arXiv:2606. 17489v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly deployed in edge-cloud inference systems to handle diverse user tasks with heterogeneous accuracy, latency, and cost profiles.

By Yin Huang, Qingsong Liu, Jie Xu

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 17

Does the Data Processing Inequality Reflect Practice? On the Utility of Low-Level Tasks

arXiv:2512. 21315v2 Announce Type: replace Abstract: The data processing inequality is an information-theoretic principle stating that the information content of a signal cannot be increased by processing the observations.

By Roy Turgeman, Tom Tirer

benchmarks

arXiv AI

Jun 17

Reversal Q-Learning

arXiv:2606. 17551v1 Announce Type: cross Abstract: Iterative generative modeling techniques, such as flow matching, provide powerful tools to model complex behaviors for effective offline reinforcement learning (RL).

By Aditya Oberai, Seohong Park, Sergey Levine

diffusion reinforcement-learning robotics benchmarks safety

arXiv AI

Jun 17

Shattering the Autoregressive Curse: Dynamic Epistemic Entropy Orchestrated Erasable Reinforcement Learning for LLMs

arXiv:2606. 17735v1 Announce Type: new Abstract: Although reinforcement learning (RL) has expanded the cognitive boundaries of large language models (LLMs), it often remains vulnerable to the autoregressive curse in long-horizon logical reasoning: small epistemic perturbations introduced early in generation can propagate irreversibly along the Markov decision process flow, triggering cascading failures that drive the reasoning trajectory toward collapse.

By Ziliang Wang, Kang An, Faqiang Qian, Jialu Cai, Cijun Ouyang, Yuhang Wang, Qibing Ren, Yichao Wu

llms reinforcement-learning benchmarks

arXiv AI

Jun 17

Detecting and Mitigating DDoS Attacks with AI: A Survey

arXiv:2503. 17867v3 Announce Type: replace-cross Abstract: Distributed Denial of Service attacks represent an active cybersecurity research problem.

By Alexandru Apostu, Silviu Gheorghe, Andrei H\^iji, Nicolae Cleju, Andrei P\u{a}tra\c{s}cu, Cristian Rusu, Radu Ionescu, Paul Irofti

benchmarks safety

arXiv AI

Jun 17

Gaussian DP for Reporting Differential Privacy Guarantees in Machine Learning

arXiv:2503. 10945v3 Announce Type: replace-cross Abstract: Current practices for reporting differential privacy (DP) guarantees for machine learning (ML) algorithms such as DP-SGD provide an incomplete and potentially misleading picture.

By Juan Felipe Gomez, Bogdan Kulynych, Georgios Kaissis, Flavio P. Calmon, Jamie Hayes, Borja Balle, Antti Honkela

computer-vision benchmarks safety

arXiv Machine Learning

Jun 17

Regularized Machine Learning for System Identification of Ship Free-Running Manoeuvres from CFD-Based Synthetic Data: A Comparative Study

arXiv:2606. 17121v1 Announce Type: cross Abstract: This study investigates supervised machine learning techniques for identifying ship hydrodynamic coefficients from CFD-generated data from free-running simulations.

By R. F. Su\'arez, J. C. Berndt, M. Abdel-Maksoud

benchmarks

arXiv Machine Learning

Jun 17

Adaptive Volumetric Mechanical Property Fields Invariant to Resolution

arXiv:2606. 18231v1 Announce Type: cross Abstract: Accurate mechanical properties (or materials) Young's modulus ($E$), Poisson's ratio ($\nu$) and density ($\rho$) are essential for reliable physics simulation of digital worlds, but most 3D assets lack this information.

By Rishit Dagli, Donglai Xiang, Vismay Modi, Xuning Yang, Gavriel State, David I. W. Levin, Maria Shugrina

llms benchmarks

arXiv Machine Learning

Jun 17

SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphs

arXiv:2606. 12867v2 Announce Type: replace Abstract: Multimodal-attributed graphs (MAGs) couple graph topology with node semantics from text, images, and other modalities.

By Zhengyu Wu, Xu Wang, Hongchao Qin, Xunkai Li, Guang Zeng, Rong-Hua Li, Guoren Wang

multimodal benchmarks safety

arXiv AI

Jun 17

When Rules Learn: A Self-Evolving Agent for Legal Case Retrieval

arXiv:2606. 17220v1 Announce Type: new Abstract: Legal case retrieval remains challenging due to the complexity of legal language and the need for precise lexical alignment between queries and relevant cases.

By Mingxu Tao, Jiawei Hu, Xian Zhou, Wenpeng Hu, Jiajun Cheng, Yunbo Cao, Zhunchen Luo, Guotong Geng

llms agents benchmarks safety

arXiv AI

Jun 17

WEQA: Wearable hEalth Question Answering with Query-Adaptive Agentic Reasoning

arXiv:2606. 18147v1 Announce Type: new Abstract: Language models are remarkably capable at medical question answering, in some cases surpassing the accuracy of general physicians.

By Yuwei Zhang, Tong Xia, Bianca Emmerich, Yu Yvonne Wu, Dimitris Spathis, Xin Liu, Daniel McDuff, Cecilia Mascolo

llms agents nlp benchmarks

arXiv AI

Jun 17

Detail++: Training-Free Detail Enhancer for Text-to-Image Diffusion Models

arXiv:2507. 17853v2 Announce Type: replace-cross Abstract: Recent advances in text-to-image (T2I) generation have led to impressive visual results.

By Lifeng Chen, Jiner Wang, Zihao Pan, Beier Zhu, Xiaofeng Yang, Chi Zhang

diffusion benchmarks safety

arXiv AI

Jun 17

SEAGym: An Evaluation Environment for Self-Evolving LLM Agents

arXiv:2606. 17546v1 Announce Type: new Abstract: Self-evolving LLM-based agents improve mainly by changing their agent harness: the structured execution layer around a base model, including prompts, memory, tools, middleware, runtime state, and the model-tool interaction loop.

By Congjie Zheng, Chuanyi Xue, Bin Liang, Jun Yang, Changshui Zhang

llms agents benchmarks

arXiv AI

Jun 17

Non-negative Elastic Net Decoding for Information Retrieval

arXiv:2606. 17910v1 Announce Type: cross Abstract: Dense retrieval has become the dominant paradigm in information retrieval, in which each document is scored against a query by the inner product of their vector embeddings, and the top-$k$ documents by score are retrieved for this query.

By Koki Okajima, Yasutoshi Ida, Tsukasa Yoshida, Yasuaki Nakamura

rag benchmarks

arXiv AI

Jun 17

PseudoBench: Measuring How Agentic Auto-Research Fuels Pseudoscience

arXiv:2606. 18060v1 Announce Type: new Abstract: As Large Language Model based agents enter autonomous scientific research, their ability to resist pseudoscience becomes increasingly important.

By Xinyang Liao, Lingyu Li, Huacan Liu, Tianle Gu, Yang Yao, Tong Zhu, Yan Teng, Yingchun Wang

llms agents benchmarks safety

arXiv AI

Jun 17

See First, Answer Later: Visual Evidence Pre-Alignment via Sufficiency-Driven RL

arXiv:2606. 17678v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) integrate strong text reasoning with visual inputs, yet their responses can be inconsistent with the underlying images, indicating ineffective utilization of visual evidence during inference.

By Yilian Liu, Sicong Leng, Guoshun Nan, Junyi Zhu, Jiayu Huang, Minghao Sun, Xuancheng Zhu, Yisong Chen, Zexian Wei, Xiaofeng Tao

llms reinforcement-learning fine-tuning multimodal benchmarks safety

arXiv AI

Jun 17

Explicit Context-Driven Neural Acoustic Modeling for High-Fidelity RIR Generation

arXiv:2509. 15210v2 Announce Type: replace-cross Abstract: Realistic sound simulation plays a critical role in many applications.

By Chen Si, Qianyi Wu, Chaitanya Amballa, Romit Roy Choudhury

benchmarks

arXiv Machine Learning

Jun 17

CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models

arXiv:2606. 17464v1 Announce Type: new Abstract: Membership inference attacks (MIAs) are a canonical way to assess a machine learning model's privacy properties.

By Jeffrey G. Wang, Jason Wang, Marvin Li, Seth Neel

llms benchmarks safety

Get the daily briefing