Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,477 stories · RSS feed

arXiv AI

Jun 30

Rethinking Role-Playing Evaluation: Anonymous Benchmarking and a Systematic Study of Personality Effects

arXiv:2603. 03915v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have shown remarkable potential in developing role-playing agents (RPAs).

By Ji-Lun Peng, Yun-Nung Chen

llms agents benchmarks

arXiv AI

Jun 30

Early Cue Precision Shapes Visual Shortcut Learning in Controlled Cue-Manipulation Benchmarks

arXiv:2606. 30344v1 Announce Type: cross Abstract: Visual classifiers can achieve high matched-distribution accuracy while relying on low-level cues that fail under conflict or suppression.

By Chanho Park, Woochan Lee, Janyeong Oh, Geongho Gong, Minshu Kim, Yeachan Kwak, Seongim Choi

robotics fine-tuning benchmarks

arXiv Machine Learning

Jun 30

Nonlinear mixture model motivated subspace clustering

arXiv:2606. 29261v1 Announce Type: new Abstract: We derive the linear union-of-subspaces (UoS) model for subspace clustering (SC) from the nonlinear mixture model (NMM) used in blind source separation (BSS) to represent a D-dimensional observation vector as an unknown multivariate nonlinear mapping of C latent variables.

By Ivica Kopriva

efficiency benchmarks

arXiv Machine Learning

Jun 30

CAN We Trust Your Results? A Cross-Dataset Study of Automotive IDS Evaluation

arXiv:2606. 30430v1 Announce Type: cross Abstract: The increasing connectivity of modern vehicles has made securing in-vehicle communication networks a critical challenge.

By Beatrix Koltai, Gergely Acs, Andras Gazdag

benchmarks

arXiv Machine Learning

Jun 30

MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory

arXiv:2606. 29788v1 Announce Type: new Abstract: When a multimodal AI agent is asked to forget a fact, current memory systems usually delete the text entry and report success.

By Kuan Wang, Chao Zhang

agents multimodal benchmarks

arXiv AI

Jun 30

Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

arXiv:2601. 05366v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly deployed as agents that invoke external tools through structured function calls.

By Zheng Luo, T Pranav Kutralingam, Ogochukwu N Okoani, Wanpeng Xu, Hua Wei, Xiyang Hu

llms agents benchmarks

arXiv AI

Jun 30

Temporal Feature Extractors in EEG Foundation Models: A Controlled Comparison Including a Pretrained Time-Series Model

arXiv:2606. 30104v1 Announce Type: new Abstract: Electroencephalography (EEG) foundation models aim to learn generalizable representations from large-scale brain recordings.

By Ay\c{s}e Bet\"ul Y\"uce, Chris Joey Leffler, Sarun Varghese, Myra Spiliopoulou, Sebastian Stober

benchmarks

arXiv Machine Learning

Jun 30

MOSAIC: Orchestrating Collaborative Knowledge Tracing with Hierarchical Semantic Alignment

arXiv:2606. 29049v1 Announce Type: new Abstract: Knowledge Tracing (KT) is important for personalized education but traditionally suffers from two key limitations: a reliance on shallow ID-based representations that neglect semantic depth and a restriction to single-granularity mastery estimation that overlooks hierarchical knowledge dependencies.

By Xinjin Li, Mengyue Wang, Yuzhen Lin, Pengbin Feng, Ziqi Sha, Yeyang Zhou, Yu Ma

llms rag benchmarks safety

arXiv AI

Jun 30

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.

By Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai, Dengfeng Liu, Qiaobo Hao, Huang Liu, Zhiguo Yang, Ji Xie, Ninglun Gu, Jin Yang, Kailai Zhang, Yelun Bao, Jun Wang

llms reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jun 30

Learning from Mistakes: Rollout-Retrieval Lifelong Policy Learning for Autonomous Driving

arXiv:2606. 30537v1 Announce Type: cross Abstract: Autonomous driving policies should be able to improve continually as deployment exposes them to increasingly diverse and long-tail traffic situations.

By Cheng Gong, Haoyang Wang, Chao Lu, Zirui Li, Jianwei Gong

agents robotics fine-tuning benchmarks

arXiv Machine Learning

Jun 30

Few-Step Boltzmann Generators via Scalable Likelihood Flow Maps

arXiv:2606. 29110v1 Announce Type: new Abstract: Recent progress in flow-based generative modeling has led to models that output high-quality samples while using only a small number of function evaluations.

By RuiKang OuYang, Hanlin Yu, Xinyue Ai, Yutong He, Nicholas M. Boffi, Pradeep Ravikumar, Jose Miguel Hernandez-Lobato, Max Simchowitz, Benjamin Kurt Miller, Omar Chehab

efficiency benchmarks

arXiv Machine Learning

Jun 30

MALOQ: Massively Accelerated Learning of Operators for Quantum Transport

arXiv:2606. 28911v1 Announce Type: new Abstract: Machine-learned (ML) operator models can be trained to predict density functional theory (DFT) Hamiltonian/density matrices at significantly reduced computational cost, thus extending electronic-structure calculations to previously unfeasible scales.

By Manasa Kaniselvan, Alexander Maeder, Denghui Lu, Alexandros Nikolaos Ziogas, Mathieu Luisier

benchmarks

arXiv Machine Learning

Jun 30

Toward an Energy-Optimized Operation of Data Centers Located in Wind Farms Using Reinforcement Learning

arXiv:2606. 30316v1 Announce Type: new Abstract: This paper studies Reinforcement Learning as an online controller for curtailment-aware workload shifting in wind-turbine-integrated high-performance computing (HPC) data centers.

By Jan Stenner, Alexander Kilian, Sebastian Peitz, Hermann de Meer

reinforcement-learning benchmarks

arXiv AI

Jun 30

RoboGaze: Evaluating Robot World Models via Structured Vision-Language Analysis

arXiv:2606. 28385v1 Announce Type: cross Abstract: Recent advances in robot world models enable synthetic video generation for embodied prediction and planning.

By Minh-Loi Nguyen, Nghiem Tuong Diep, Hung Khang Nguyen, Minh Le, Doanh Le Thien, Hoang H. Tran, Dung D. Le, Vu N. Duong, Daniel Sonntag, An Thai Le, Duy Minh Ho Nguyen, Vien Anh Ngo, Tran Van Nhiem

llms agents diffusion robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 30

Prototype Latent World Model Replay for Class-Incremental Learning

arXiv:2606. 29465v1 Announce Type: new Abstract: Class-incremental learning requires a model to learn new classes while preserving decision regions for old ones.

By Weizhi Nie, Hui Wang, Weijie Wang, Yuting Su

fine-tuning benchmarks

arXiv AI

Jun 30

TAR: Temporal Anchor-Constrained Reasoning for Video Temporal Grounding

arXiv:2508. 07683v2 Announce Type: replace-cross Abstract: Video Temporal Grounding (VTG) aims to localize specific video segments corresponding to natural language queries.

By Chaohong Guo, Xun Mo, Yongwei Nie, Fei Ma, Xuemiao Xu, Chengjiang Long

llms agents reinforcement-learning multimodal benchmarks safety

arXiv Machine Learning

Jun 30

Improved Predictive Performance and Interpretability for Mesomorphic Neural Networks Using Local Fidelity Regularization

arXiv:2606. 29951v1 Announce Type: new Abstract: Interpretable Mesomorphic Neural Networks (IMNs) offer a promising framework that combines the predictive power of deep neural networks with the interpretability of linear models.

By Hugo L. Hammer, Vajira Thambawita, Kristoffer Herland Hellton, P{\aa}l Halvorsen

benchmarks safety

arXiv Machine Learning

Jun 30

Local-Minima-Preserving Continuous Relaxation of Ising Problems

arXiv:2606. 30333v1 Announce Type: cross Abstract: The generalized Ising problem captures a broad spectrum of hard combinatorial problems, including MAX-CUT, Number Partitioning (NPP), and Maximum Independent Set.

By Debraj Banerjee, Santanu Mahapatra, Kunal N. Chaudhury

benchmarks

arXiv Machine Learning

Jun 30

GLACIER: Rethinking Mass Spectrum Prediction as an Object Detection Problem

arXiv:2606. 29161v1 Announce Type: new Abstract: Predicting tandem mass spectra (MS/MS) from molecular structures represents a central task in analytical chemistry with direct relevance to clinical metabolomics, systems biology, and adjacent disciplines.

By Rui-Xi Wang, Runzhong Wang, Connor W. Coley

llms computer-vision fine-tuning benchmarks

arXiv Machine Learning

Jun 30

PCGD: Physics-Guided Conditional Graph Diffusion for TCAD Device Simulation

arXiv:2606. 29272v1 Announce Type: new Abstract: Technology computer-aided design (TCAD) semiconductor device simulation is fundamentally constrained by the high computational cost of iteratively solving coupled drift-diffusion equations.

By Yihan Zhang, Zhiteng Zhang, Kun Chen, Chen Wang

diffusion fine-tuning benchmarks

Get the daily briefing