Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,477 stories · RSS feed

arXiv AI

Jun 30

RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning

arXiv:2606. 29867v1 Announce Type: cross Abstract: Deep Reinforcement Learning (DRL) has achieved significant success in robotics and autonomous systems, yet remains vulnerable to adversarial perturbations that can severely degrade performance.

By Adithya Mohan, Daniel Kriegl, Torsten Sch\"on

agents reinforcement-learning robotics benchmarks safety

arXiv AI

Jun 30

Selective Memory Retention for Long-Horizon LLM Agents

arXiv:2606. 29178v1 Announce Type: new Abstract: When does retention matter for memory-augmented LLM agents?

By Pranath Reddy

llms rag agents benchmarks

arXiv AI

Jun 30

HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data

arXiv:2606. 29784v1 Announce Type: cross Abstract: Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity.

By Xinrui Ruan, Zhenyu Zhao, Waverly Wei, Yueshan Zhang, Zeyu Zheng, Sui Huang, Jingshen Wang

benchmarks safety

arXiv Machine Learning

Jun 30

How Far Do On-Prem Open LLMs Get on Text-to-SQL? A Cross-Family Size x Technique Frontier on BIRD

arXiv:2606. 29733v1 Announce Type: cross Abstract: Organizations that cannot send data to a cloud API increasingly ask: how good is Text-to-SQL if the model must run on-premises on open weights, and which popular accuracy "recipes" are worth their compute?

By Vladimir Beskorovainyi

llms rag benchmarks data-engineering

arXiv Machine Learning

Jun 30

Diffusion Fine-tuning with Rewarded Moment Matching Distillation

arXiv:2606. 30414v1 Announce Type: new Abstract: Distillation and Reinforcement Learning (RL) fine-tuning are the primary pillars of diffusion post-training.

By Alexis Jacq, Guillaume Couairon, Valentin De Bortoli, Quentin Berthet, Arnaud Doucet, Romuald Elie

diffusion reinforcement-learning fine-tuning efficiency benchmarks

arXiv AI

Jun 30

FlatLands: Generative Floormap Completion From a Single Egocentric View

arXiv:2603. 16016v2 Announce Type: replace-cross Abstract: A single egocentric image typically captures only a small portion of the floor, yet a complete metric traversability map of the surroundings would better serve applications such as indoor navigation.

By Subhransu S. Bhattacharjee, Dylan Campbell, Rahul Shome

robotics benchmarks

arXiv AI

Jun 30

Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework

arXiv:2606. 29808v1 Announce Type: cross Abstract: Chart data extraction, which reverse-engineers data tables from chart images, is essential for reproducibility, analysis, retrieval, and redesign.

By Yuchen He, Peizhi Ying, Liqi Cheng, Kuilin Peng, Yuan Tian, Dazhen Deng, Yingcai Wu

llms multimodal benchmarks

arXiv AI

Jun 30

Inference-Time Diversity in RL-Trained Lean Theorem Provers: A Diagnostic Study

arXiv:2601. 16172v3 Announce Type: replace Abstract: RL-trained Lean theorem provers mode-collapse at inference time: on miniF2F-test with DeepSeek-Prover-V1.

By Zachary Burton

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jun 30

HMARS: A Hierarchical Multi-Agent Memory System for Long-Context Reasoning

arXiv:2606. 28349v1 Announce Type: cross Abstract: Long-context reasoning requires models to access, retrieve, and integrate evidence scattered across documents, dialogues, and accumulated interaction histories.

By Zeju Li, Ziyang Zheng, Yizhou Zhou, Qiang Xu

rag agents benchmarks

arXiv AI

Jun 30

LoRA-Tuned Large Language Models for Dementia Detection via Multi-View Speech-Derived Features

arXiv:2606. 28445v1 Announce Type: cross Abstract: Early detection of dementia enables timely intervention, and reflecting cognitive impairment, spontaneous speech offers a non-invasive screening modality.

By Jonghyeon Park, Olivier Jiyoun Jung, Myungwoo Oh

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 30

Dual-Flow Reinforcement Learning with State-Aware Exploration

arXiv:2606. 29820v1 Announce Type: cross Abstract: In complex continuous-control reinforcement learning tasks, multimodal optimal actions often coincide with uncertain, multimodal return distributions, making reliable value estimation and multimodal exploration challenging.

By Qijun Li, Zheng Fu, Qi Song, Yifei He, Weitao Zhou, Kun Jiang, Diange Yang

diffusion reinforcement-learning multimodal benchmarks

arXiv AI

Jun 30

DRIFT: Difficulty Routing Self-DIstillation with Rhythm-Gated Exploration and Success BuFfer Training

arXiv:2606. 30345v1 Announce Type: cross Abstract: Enabling large language models to achieve stable self-improvement without external expert supervision remains a central challenge in complex reasoning tasks.

By Haisen Luo, Yiwei Liu, Haoning Wang, Dan Liu, Junxi Yin, Haotian Wang, Lei Zhang, Xiaoyu Tian, Shuaiting Chen, Yuansheng Song, Baoyan Guo, Xiongfei Yan, Bolan Yang, Chengwei Liu, Ming Cui, Jiong Chen

llms reinforcement-learning efficiency benchmarks

arXiv AI

Jun 30

DuoMem: Towards Capable On-Device Memory Agents via Dual-Space Distillation

arXiv:2606. 29961v1 Announce Type: cross Abstract: Large Language Model (LLM)-based agents can solve complex procedural tasks by interacting with environments over multiple turns, but this ability typically depends on large models, long contexts, and repeated inference calls.

By Peyman Hosseini, Ondrej Bohdal, Ahmed Alajrami, Andrea Maracani, Ignacio Castro, Matthew Purver, Mete Ozay, Savas Ozkan, Taha Ceritli

llms agents robotics fine-tuning efficiency benchmarks

arXiv AI

Jun 30

SEVA: Self-Evolving Verification Agent with Process Reward for Fact Attribution

arXiv:2606. 29713v1 Announce Type: cross Abstract: Hallucination is the reliability bottleneck for LLM-based agents, and fact attribution verifiers are the last line of defense -- yet today's verifiers emit only opaque binary labels, leaving agents unable to self-correct and operators unable to audit.

By Aojie Yuan, Yi Nian, Haiyue Zhang, Zijian Su, Yue Zhao

llms agents reinforcement-learning benchmarks safety

arXiv AI

Jun 30

MotionAtlas: Detailed Region Captioning for Motion-Centric Videos

arXiv:2606. 29531v1 Announce Type: cross Abstract: We propose MotionAtlas, a system for detailed captioning of motion-centric videos, comprising (1) a dedicated human-annotated benchmark, (2) a scalable, high-quality pipeline to construct training samples, and (3) a family of powerful Video-MLLMs.

By Weisong Liu, Haochen Wang, Kuan Gao, Yuhao Wang, Yikang Zhou, Zhongwei Ren, Jacky Mai, Anna Wang, Yanwei Li, Jason Li, Zhaoxiang Zhang

benchmarks safety data-engineering

arXiv Machine Learning

Jun 30

Nonlinear mixture model motivated subspace clustering

arXiv:2606. 29261v1 Announce Type: new Abstract: We derive the linear union-of-subspaces (UoS) model for subspace clustering (SC) from the nonlinear mixture model (NMM) used in blind source separation (BSS) to represent a D-dimensional observation vector as an unknown multivariate nonlinear mapping of C latent variables.

By Ivica Kopriva

efficiency benchmarks

arXiv Machine Learning

Jun 30

CAN We Trust Your Results? A Cross-Dataset Study of Automotive IDS Evaluation

arXiv:2606. 30430v1 Announce Type: cross Abstract: The increasing connectivity of modern vehicles has made securing in-vehicle communication networks a critical challenge.

By Beatrix Koltai, Gergely Acs, Andras Gazdag

benchmarks

arXiv Machine Learning

Jun 30

MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory

arXiv:2606. 29788v1 Announce Type: new Abstract: When a multimodal AI agent is asked to forget a fact, current memory systems usually delete the text entry and report success.

By Kuan Wang, Chao Zhang

agents multimodal benchmarks

arXiv AI

Jun 30

Temporal Feature Extractors in EEG Foundation Models: A Controlled Comparison Including a Pretrained Time-Series Model

arXiv:2606. 30104v1 Announce Type: new Abstract: Electroencephalography (EEG) foundation models aim to learn generalizable representations from large-scale brain recordings.

By Ay\c{s}e Bet\"ul Y\"uce, Chris Joey Leffler, Sarun Varghese, Myra Spiliopoulou, Sebastian Stober

benchmarks

arXiv AI

Jun 30

Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning

arXiv:2508. 09883v2 Announce Type: replace-cross Abstract: Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.

By Xiaojun Wu, Xiaoguang Jiang, Huiyang Li, Jucai Zhai, Dengfeng Liu, Qiaobo Hao, Huang Liu, Zhiguo Yang, Ji Xie, Ninglun Gu, Jin Yang, Kailai Zhang, Yelun Bao, Jun Wang

llms reinforcement-learning fine-tuning efficiency benchmarks

Get the daily briefing