Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,285 stories · RSS feed

arXiv Machine Learning

Jul 1

Sequential RC-TGAN: Generating Relational Time Series with Spectral Envelope Loss

arXiv:2606. 31904v1 Announce Type: new Abstract: The generation of synthetic relational databases often involves modeling complex temporal dynamics, such as transaction logs or event sequences.

By Mohamed Gueye, Yazid Attabi, Manuel Morales, Maxime Dumas

benchmarks

arXiv Machine Learning

Jul 1

Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks

arXiv:2606. 31700v1 Announce Type: new Abstract: Biological neural circuits obey Dale's principle: each neuron's synapses are uniformly excitatory or inhibitory.

By Yutaro Yamada, Luca Grillotti, Rujikorn Charakorn, Sebastian Risi, David Ha, Robert Tjarko Lange

diffusion reinforcement-learning benchmarks safety

arXiv AI

Jul 1

Adaptive Cluster-First Route-Second Decomposition for Industrial-Scale Vehicle Routing

arXiv:2606. 31820v1 Announce Type: new Abstract: Large-scale capacitated vehicle routing problems (CVRPs) are commonly addressed using cluster-first route-second (CFRS) approaches that split a routing instance into smaller, computationally tractable subproblems.

By Oguzhan Karaahmetoglu (Carnegie Mellon University), Hyong Kim (Carnegie Mellon University)

llms benchmarks

arXiv AI

Jul 1

Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs

arXiv:2606. 32032v1 Announce Type: cross Abstract: Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes.

By Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, Arman Cohan

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 1

ASR-Agnostic Multimodal Spectrotemporal Modeling for Early Dementia Detection

arXiv:2606. 30646v1 Announce Type: cross Abstract: Speech recruits the same executive, attentional, and working memory processes underlying instrumental activities of daily living, or IADLs, providing a non-invasive proxy for cognitive assessment.

By Chukwuemeka Ugwu, Oluwafemi Richard Oyeleke

llms rag multimodal benchmarks

arXiv AI

Jul 1

Qualified Educational Capacity Planning under Heterogeneous Student Support Needs: A Synthetic Benchmark and Decision-Support Framework

arXiv:2606. 30650v1 Announce Type: cross Abstract: Educational support services often face a qualified-capacity problem: staff time is scarce, qualifications decay, new support needs can appear before anyone is prepared for them, and training consumes the same hours needed by current students.

By Carlos Eduardo Sanoja, Oscar Enrique Moreno Mayz

benchmarks safety

arXiv AI

Jul 1

Cross-Modal Hierarchical Fusion for from Multi-Sensor Ground Observation

arXiv:2606. 30647v1 Announce Type: cross Abstract: Dense volumetric reconstruction of cloud microphysical fields from sparse ground-based instruments remains an open problem, largely because the available measurements are heterogeneous in both modality and spatial coverage.

By Xinze Zhang

multimodal benchmarks

arXiv AI

Jul 1

Surrogate-Gated Generation and Foundation-Model Embeddings for Bayesian Materials Design

arXiv:2606. 28578v1 Announce Type: cross Abstract: Closed-loop materials discovery iterates between proposing candidate structures and evaluating their properties, and property evaluation dominates the cost.

By Sk Md Ahnaf Akif Alvi, Jan Janssen, Danny Perez, Douglas Allaire, Raymundo Arroyave

rag diffusion reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 1

REMSA: Foundation Model Selection for Remote Sensing via a Constraint-Aware Agent

arXiv:2511. 17442v3 Announce Type: replace-cross Abstract: Foundation Models (FMs) are increasingly integrated into remote sensing (RS) pipelines.

By Binger Chen, Tacettin Emre B\"ok, Behnood Rasti, Volker Markl, Beg\"um Demir

llms agents computer-vision nlp multimodal benchmarks

arXiv AI

Jul 1

Evo-PI: Aligning Medical Reasoning via Evolving Principle-Guided Supervision

arXiv:2606. 31800v1 Announce Type: new Abstract: Despite recent progress, the reasoning capabilities of large multimodal language models (MLLMs) remain fundamentally constrained by static supervision, where fixed prompts, rules, or reward models provide non-adaptive guidance throughout training.

By Xianda Zheng, Huan Gao, Meng-Fen Chiang, Michael Witbrock, Kaiqi Zhao, Shangyang Li

llms nlp multimodal benchmarks safety

arXiv AI

Jul 1

The HydroGym Reinforcement Learning Platform for Fluid Dynamics

arXiv:2512. 17534v2 Announce Type: replace-cross Abstract: Modeling and controlling fluids is critical across science and engineering.

By Christian Lagemann, Sajeda Mokbel, Miro Gondrum, Mario R\"uttgers, Yuning Wang, Pol Su\'arez, Ludger Paehler, Deniz A. Bezgin, Aaron B. Buhendwa, Jared L. Callaham, Samuel Ahnert, Nicholas Zolman, Xiao Shao, Jean-Christophe Loiseau, Nikolaus Adams, Matthias Meinke, Wolfgang Schr\"oder, Kai Lagemann, Esther Lagemann, Ricardo Vinuesa, Steven L. Brunton

agents reinforcement-learning robotics benchmarks

arXiv AI

Jul 1

ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences

arXiv:2602. 11354v3 Announce Type: replace Abstract: The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers.

By Bang Nguyen, Dominik So\'os, Qian Ma, Rochana R. Obadage, Zack Ranjan, Sai Koneru, Timothy M. Errington, Shakhlo Nematova, Sarah Rajtmajer, Jian Wu, Meng Jiang

llms agents benchmarks

arXiv AI

Jul 1

Improving Certified Robustness via Adversarial Distillation

arXiv:2606. 31653v1 Announce Type: cross Abstract: Certified training aims to produce models whose predictions can be formally verified against adversarial perturbations, typically by optimising upper bounds on the worst-case loss over an allowed perturbation set.

By Matteo Melis, Jesus Martinez Del Rincon, Vishal Sharma

efficiency benchmarks safety

arXiv AI

Jul 1

A swap-adversarial framework for improving domain generalization in electrocorticography-based Parkinson's disease classification

arXiv:2602. 10528v2 Announce Type: replace-cross Abstract: We propose a novel swap-adversarial framework that mitigates high inter-subject variability and the high-dimensional low-sample-size problem in electrocorticography (ECoG) data.

By Seongwon Jin, Hanseul Choi, Sunggu Yang, Sungho Park, Jibum Kim

benchmarks safety

arXiv AI

Jul 1

DeXposure-FM: A Time-series, Graph Foundation Model for Credit Exposures and Stability on Decentralized Financial Networks

arXiv:2602. 03981v2 Announce Type: replace-cross Abstract: Credit exposure in Decentralized Finance (DeFi) is often implicit and token-mediated, creating a dense web of inter-protocol dependencies.

By Aijie Shu, Wenbin Wu, Gbenga Ibikunle, Fengxiang He

benchmarks

arXiv AI

Jul 1

AgRefactor: Self-Evolving Agentic Workflow for HLS Compatibility and Performance

arXiv:2606. 30949v1 Announce Type: new Abstract: High-Level Synthesis (HLS) provides a fast path from concepts to silicon, but converting real-world software into synthesizable HLS code remains challenging due to restrictive language support and the gap between software and hardware programming practices.

By Yang Zou, Zijian Ding, Yizhou Sun, Jason Cong

llms agents benchmarks

arXiv AI

Jul 1

Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference

arXiv:2606. 31903v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) increasingly process long visual-token sequences, increasing the overall inference computation.

By Zhaoyang Luo, Runmin Dong, Miao Yang, Fan Wei, Yushan Lai, Bin Luo, Haohuan Fu

llms multimodal benchmarks

arXiv AI

Jul 1

BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations

arXiv:2606. 30700v1 Announce Type: cross Abstract: Self-supervised learning enables audio representations that transfer across domains and tasks.

By Ludovic K. Tuncay (IRIT-SAMoVA), Etienne Labb\'e (IRIT-SAMoVA), Thomas Pellegrini (IRIT-SAMoVA)

llms benchmarks

arXiv Machine Learning

Jul 1

Relational and Sequential Conformal Inference for Energy Time Series over Graphs via Foundation Models

arXiv:2606. 31804v1 Announce Type: new Abstract: Accurate energy demand forecasting is essential for the reliable operation and planning of modern sustainable energy systems.

By Keivan Faghih Niresi, Alice Cicirello, Olga Fink

llms benchmarks safety

arXiv AI

Jul 1

RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora

arXiv:2604. 19047v2 Announce Type: replace-cross Abstract: Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate on corpora such as financial reports, legal codes, and patents, where information is highly redundant and documents exhibit strong inter-document similarity.

By Hanjun Cho, Jay-Yoon Lee

llms rag benchmarks

Get the daily briefing