Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

20,687 stories · RSS feed

arXiv Machine Learning

Jun 24

Zero-Shot Neural Priors for Generalizable Cross-Subject and Cross-Task EEG Decoding

arXiv:2606. 23706v1 Announce Type: cross Abstract: The development of generalizable electroencephalography (EEG) decoding models is essential for robust brain-computer interfaces (BCI) and objective neural biomarkers in mental health.

By Baimam Boukar Jean Jacques, Brandone Fonya, Nchofon Tagha Ghogomu, Pauline Nyaboe, Kipngeno Koech

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 24

EERLoss: A Novel Loss Function for Training Deep Biometric Models. A Case Study in Keystroke Dynamics

arXiv:2606. 24586v1 Announce Type: cross Abstract: Deep learning approaches to biometric verification are commonly trained by optimizing indirect objectives, creating a misalignment between the optimization process and the primary evaluation metric, typically the Equal Error Rate (EER).

By Nahuel Gonzalez, Marta Robledo-Moreno, Ivan DeAndres-Tame, Ruben Vera-Rodriguez, Ruben Tolosana

benchmarks

arXiv Machine Learning

Jun 24

LoMime: Query-Efficient Membership Inference using Model Extraction in Label-Only Settings

arXiv:2602. 18934v2 Announce Type: replace Abstract: Membership inference attacks (MIAs) threaten the privacy of machine learning models by revealing whether a specific data point was used during training.

By Abdullah Caglar Oksuz, Anisa Halimi, Erman Ayday

benchmarks safety

arXiv AI

Jun 24

LaGO: Latent Action Guidance for Online Reinforcement Learning

arXiv:2606. 24669v1 Announce Type: new Abstract: Large language models (LLMs) have shown strong potential for planning and sequential decision-making, but prior work often relies on using them as direct controllers, which requires precise action generation and can be unreliable in practice.

By Kuan-Yen Liu, Ren-Jyun Huang, Ti-Rong Wu

llms reinforcement-learning robotics benchmarks

arXiv AI

Jun 24

OpenThoughts-Agent: Data Recipes for Agentic Models

arXiv:2606. 24855v1 Announce Type: new Abstract: Agentic language models dramatically expand the applications of AI yet little is publicly known about how to curate training data for broadly capable agents.

By Negin Raoof, Richard Zhuang, Marianna Nezhurina, Etash Guha, Atula Tejaswi, Ryan Marten, Charlie F. Ruan, Tyler Griggs, Alexander Glenn Shaw, Hritik Bansal, E. Kelly Buchanan, Artem Gazizov, Reinhard Heckel, Chinmay Hegde, Sankalp Jajee, Daanish Khazi, Emmanouil Koukoumidis, Xiangyi Li, Hange Liu, Shlok Natarajan, Harsh Raj, Nicholas Roberts, Ethan Shen, Nishad Singhi, Michael Siu, Ashima Suvarna, Hanwen Xing, Patrick Yubeaton, Robert Zhang, Leon Liangyu Chen, Xiaokun Chen, Steven Dillmann, Saadia Gabriel, Xunyi Jiang, Anurag Kashyap, Boxuan Li, Yein Park, Minh Pham, Sujay Sanghavi, Lin Shi, Ke Sun, Yixin Wang, Zhiwei Xu, Erica Zhang, Siyan Zhao, Wanjia Zhao, Jenia Jitsev, Alex Dimakis, Benjamin Feuer, Ludwig Schmidt

llms agents fine-tuning benchmarks data-engineering

arXiv AI

Jun 24

Co-occurring associated retained concepts in Diffusion Unlearning

arXiv:2606. 24192v1 Announce Type: cross Abstract: Unlearning has emerged as a key technique to mitigate harmful content generation in diffusion models.

By Miso Kim, Georu Lee, Yunji Kim, Hoki Kim, Jinseong Park, Woojin Lee

diffusion benchmarks

arXiv AI

Jun 24

Reinforcement Learning Towards Broadly and Persistently Beneficial Models

arXiv:2606. 24014v1 Announce Type: new Abstract: As AI systems are deployed across increasingly diverse and high-stakes settings, model alignment must generalize beyond the tasks and domains seen during training.

By Akshay V. Jagadeesh, Rahul K. Arora, Khaled Saab, Ali Malik, Mikhail Trofimov, Foivos Tsimpourlas, Johannes Heidecke, Karan Singhal

llms reinforcement-learning fine-tuning benchmarks safety

arXiv AI

Jun 24

When Helpfulness Overrides Causal Caution: Context-Dependent Suppression and Recovery in LLMs

arXiv:2606. 24370v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into decision-support roles in business and policy contexts.

By Hiroshi Okumura

llms agents benchmarks

arXiv AI

Jun 24

DTT-BSR+: A Generative-Regression Cascade for Music Source Restoration

arXiv:2606. 24127v1 Announce Type: cross Abstract: Music source restoration (MSR) requires jointly addressing source unmixing and the inversion of non-linear production effects.

By Youran Ni, Shihong Tan, Yuzhu Wang, Gongping Huang

benchmarks

arXiv AI

Jun 24

Are Safety Guarantees in Neural Networks Safe? How to Compute Trustworthy Robustness Certifications

arXiv:2606. 23858v1 Announce Type: cross Abstract: A primary challenge in AI safety is the existence of adversarial examples -- slightly distorted inputs that cause a neural network (NN) to misclassify.

By Merkouris Papamichail, Konstantinos Varsos, Giorgos Flouris, Jo\~ao Marques-Silva

benchmarks safety

arXiv Machine Learning

Jun 24

ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs

arXiv:2510. 04767v2 Announce Type: replace Abstract: While most autoregressive LLMs are constrained to one-by-one decoding, diffusion LLMs (dLLMs) have attracted growing interest for their potential to dramatically accelerate inference through parallel decoding.

By Wonjun Kang, Kevin Galim, Seunghyuk Oh, Minjae Lee, Yuchen Zeng, Shuibai Zhang, Coleman Hooper, Yuezhou Hu, Hyung Il Koo, Nam Ik Cho, Kangwook Lee

llms diffusion benchmarks

arXiv Machine Learning

Jun 24

Generating adversarial inputs for a graph neural network model of AC power flow

arXiv:2602. 17975v2 Announce Type: replace Abstract: This work formulates and solves optimization problems to generate input points that yield high errors between a neural network's predicted AC power flow solution and solutions to the AC power flow equations.

By Robert Parker

benchmarks safety

arXiv Machine Learning

Jun 24

L3Cube-MahaPOS: A Marathi Part-of-Speech Tagging Dataset and BERT Models

arXiv:2606. 24825v1 Announce Type: cross Abstract: Part-of-Speech (POS) tagging is a foundational NLP task underpinning machine translation, information extraction, and syntactic parsing.

By Hariom Ingle, Ronit Ghode, Ishwari Gondkar, Jidnyasa Harad, Raviraj Joshi

llms nlp benchmarks

arXiv Machine Learning

Jun 24

Machine Learning Modeling for Real-Time Melt Pool Monitoring in Laser Powder Bed Fusion Additive Manufacturing: A Hybrid Approach

arXiv:2606. 23851v1 Announce Type: new Abstract: This work investigates the implementation of artificial intelligence and machine learning (AI/ML) for real-time monitoring in laser powder bed fusion (LPBF) additive manufacturing.

By Inioluwa Emmanuel, Zhuo Yang, Ho Yeung, Xinyao Zhang

rag computer-vision fine-tuning efficiency benchmarks

arXiv Machine Learning

Jun 24

NeuroSonic: Conditional Flow Matching for EEG-to-Speech Reconstruction

arXiv:2606. 24087v1 Announce Type: new Abstract: Reconstructing continuous speech from scalp electroencephalography (EEG) remains fundamentally challenging.

By Wenhao Gao, Yifan Wang, Yijia Ma, Carl Yang, Wen Li, Chenyu You

llms diffusion benchmarks

arXiv Machine Learning

Jun 24

Holistic Data Scheduler for LLM Pre-training via Multi-Objective Reinforcement Learning

arXiv:2606. 24133v1 Announce Type: new Abstract: The composition of training data, governed by the diversity of sources and their mixing strategy, is a cornerstone of Large Language Model (LLM) pre-training.

By Chenhao Dang, Jing Ma, Mingjie Liao

llms reinforcement-learning benchmarks

arXiv AI

Jun 24

Mind the Heads: Topological Representation Alignment for Multimodal LLMs

arXiv:2606. 23885v1 Announce Type: cross Abstract: Representation alignment has emerged as an effective approach to improve Multimodal Large Language Models (MLLMs) by regularizing their internal representations toward those of an external vision encoder.

By Davide Caffagni, Alberto Compagnoni, Federico Melis, Sara Sarto, Pier Luigi Dovesi, Mark Granroth-Wilding, Marcella Cornia, Lorenzo Baraldi

llms multimodal benchmarks safety

arXiv AI

Jun 24

Deciphering Fingerprints of 3D Molecular Surfaces for Accurate Epitope Prediction

arXiv:2606. 23830v1 Announce Type: cross Abstract: Molecular surfaces encode the geometric and physicochemical patterns that determine antibody-antigen recognition, central to epitope prediction.

By Fang Wu, Weihao Xuan, Jure Leskovec, Yejin Choi, Li Erran Li

llms benchmarks

arXiv Machine Learning

Jun 24

Debate2Create: Robot Co-design via Multi-Agent LLM Debate

arXiv:2510. 25850v3 Announce Type: replace-cross Abstract: We introduce Debate2Create (D2C), a multi-agent LLM framework that formulates robot co-design as structured, iterative debate grounded in physics-based evaluation.

By Kevin Qiu, Marek Cygan

llms agents reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jun 24

FuseSampleAgg: One-Pass Neighborhood Estimation for Budgeted Knowledge-Graph Refresh and Validation

arXiv:2511. 13645v2 Announce Type: replace Abstract: Operational knowledge-graph (KG) pipelines in networking and cybersecurity increasingly need to refresh embeddings under strict time, memory, and audit budgets, especially as curated feeds and LLM-assisted extraction accelerate KG updates.

By Aleksandar Stankovi\'c, Haoran Du, Xinming Wang

llms rag benchmarks

Get the daily briefing