Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,065 stories · RSS feed

arXiv AI

Jul 24

The Active Ingredient in Muon's Grokking

arXiv:2607. 20512v1 Announce Type: cross Abstract: The Muon optimizer reaches the grokking threshold on modular arithmetic faster than AdamW.

By Yufeng Wang

rag benchmarks

arXiv AI

Jul 24

ArbiGraph: Arbitrarily Scalable Verifiable Task Graphs for Evaluating Context Management

arXiv:2607. 20764v1 Announce Type: new Abstract: We introduce ARBIGRAPH, a benchmark generator for evaluating whether tool-assisted language agents can retain, update, compose, and discard task-relevant context across extended reasoning workflows.

By Pavel Golikov, Evgenii Opryshko, Gennady Pekhimenko, Mark C. Jeffrey

agents benchmarks

arXiv Machine Learning

Jul 24

MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems

arXiv:2509. 22047v3 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO) has been shown to be an effective algorithm when an accurate reward model is available.

By Yuki Ichihara, Yuu Jinnai, Tetsuro Morimura, Mitsuki Sakamoto, Ryota Mitsuhashi, Eiji Uchibe

nlp reinforcement-learning benchmarks

arXiv AI

Jul 24

GPE: Evaluating Robust Evidence Aggregation for Fact Verification under Controllable GEO-Style Poisoning

arXiv:2607. 20730v1 Announce Type: cross Abstract: Large language models increasingly use search tools to retrieve up-to-date information, introducing a new attack surface in which retrieved documents can be manipulated.

By Zhaoqi Wang, Zijian Zhang, Xiaomei Yuan, Pengtao Kou, Jiamou Liu, Zhen Li, Liehuang Zhu

llms benchmarks safety

arXiv AI

Jul 24

Attention-based Experience Replay Framework for Continual Learning of Agnostic Time Series Forecasting Models

arXiv:2607. 20493v1 Announce Type: new Abstract: Deep learning has led to remarkable progress in artificial intelligence, particularly in robotics, imaging and sound processing.

By Quentin Besnard (RFAI), Nicolas Ragot (RFAI)

robotics benchmarks

arXiv AI

Jul 24

Beyond Liars' Bench: The Impact of Lie Typology, Depth, and Sparsity on Deception Detection in LLMs

arXiv:2607. 20479v1 Announce Type: new Abstract: Training probes to detect deceptive outputs from large language models is still an open problem.

By Amr Moustafa, Max Feser, Florian Mai

llms efficiency benchmarks

arXiv Machine Learning

Jul 24

Multimodal CoLRAG-TF: Triple-Filtered Retrieval for Complex PDFs

arXiv:2607. 20517v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) over heterogeneous PDF collections remains challenging due to multimodal content, domain-specific terminology, and the need for multi-hop reasoning across dispersed evidence.

By Takato Yasuno

llms rag computer-vision multimodal benchmarks safety

arXiv AI

Jul 24

DC-Leap: Training-Free Acceleration of dLLMs via Draft-Guided Contiguous Leaping Decoding

arXiv:2607. 20467v1 Announce Type: new Abstract: While parallel decoding is central to the efficiency of Diffusion Large Language Models (dLLMs), current strategies are often hindered by overly conservative confidence thresholds.

By Yanhua Jiao, Tianyi Wu, Xiaoxi Sun, Yulin Li, HuiLing Zhen, Libo Qin, Baotian Hu, Zhuotao Tian, Min Zhang

llms diffusion benchmarks

arXiv AI

Jul 24

Stochastic Sampling is Epistemically Shallow: The Dimensionality Gap Between Temperature Variation and Model Diversity in LLMs

arXiv:2607. 20464v1 Announce Type: new Abstract: When a language model gives different answers on repeated runs, does that variation reveal what it does not know?

By Izhar Ali

llms benchmarks

arXiv AI

Jul 24

Marking the Wrong Symptoms: Evaluating LLM Watermarks in Medical Texts

arXiv:2607. 20462v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into clinical workflows, stressing the need for reliable traceability of model-generated output with watermarking.

By Melanie Rieff, Robin Staab, Thibaud Gloaguen, Stefan Hegselmann, Martin Vechev

llms multimodal benchmarks safety

arXiv AI

Jul 24

Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation

arXiv:2607. 20478v1 Announce Type: cross Abstract: Infrastructure-as-Code (IaC) generation from natural language requires satisfying provider schemas, dependency planning, and organizational policy constraints, not merely producing syntactically plausible configurations.

By Mohamed Jouini

llms rag agents benchmarks

arXiv AI

Jul 24

THOR: A Theta-Gamma Hierarchical Oscillatory Reasoning Framework for Multi-hop QA

arXiv:2607. 20459v1 Announce Type: cross Abstract: Multi-hop question answering requires retrieving and integrating evidence from multiple contexts.

By Ziyang Ling, Ronald X. Xu, Mingzhai Sun

nlp benchmarks

arXiv AI

Jul 24

Multimodal Pretraining for Generalizable EEG Representation Learning

arXiv:2607. 21384v1 Announce Type: new Abstract: Electroencephalography (EEG) models used for epilepsy are often limited to specific datasets and tasks.

By Targol Bakhtiarvand, Jugal Kalita, Adham Atyabi

llms rag computer-vision fine-tuning multimodal benchmarks safety

arXiv AI

Jul 24

V-DEAL: Diagnosing Video Safety De-Calibration as an Understanding-Refusal Coupling Failure

arXiv:2607. 21151v1 Announce Type: new Abstract: As Video Large Language Models are increasingly deployed in real-world applications, ensuring their safety alignment has become critical.

By Zhetong Zhang, Honghao Fu, Miao Xu, Yiwei Wang, Yujun Cai

llms fine-tuning benchmarks safety

arXiv Machine Learning

Jul 24

Tackling Heterogeneity in Federated Learning via Variance-Reduced Boltzmann Sampling within Homogeneous Social Coalitions

arXiv:2506. 02897v3 Announce Type: replace Abstract: Federated Learning (FL) enables privacy-preserving collaborative model training, but its effectiveness is often limited by client data heterogeneity.

By Alessandro Licciardi, Roberta Raineri, Anton Proskurnikov, Lamberto Rondoni, Lorenzo Zino

benchmarks safety

arXiv AI

Jul 24

Expert Behavior Prior Reinforcement Learning

arXiv:2607. 21302v1 Announce Type: new Abstract: Behavior prior reinforcement learning (BPRL) has emerged as a promising paradigm to improve sample efficiency in online reinforcement learning (RL) by leveraging policy priors derived from offline demonstrations.

By Gong Gao, Weidong Zhao, Xianhui Liu, Ning Jia

agents reinforcement-learning robotics benchmarks

arXiv Machine Learning

Jul 24

DART: A Degradation-Aware Recurrent Transformer for Archival Film Restoration

arXiv:2607. 21219v1 Announce Type: cross Abstract: Archival film restoration is a challenging problem because historical footage contains compound degradations such as scratches, dust, blur, noise, flicker, and photometric aging, while clean reference videos are unavailable.

By Miko{\l}aj Jastrz\k{e}bski, Wojciech Koz{\l}owski, Kamil Adamczewski

llms benchmarks

arXiv Machine Learning

Jul 24

Chronofy: A Temporal-Logical Decay Architecture for Information Validity in Time-Aware Retrieval-Augmented Generation

arXiv:2607. 20560v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) systems retrieve and integrate external knowledge to ground large language model (LLM) outputs.

By Muntaser Syed, Marius Silaghi, Sheikh Abujar, Sharun Akter

llms rag benchmarks safety

arXiv AI

Jul 24

Generative Artificial Intelligence in Bioinformatics: A Systematic Review of Models, Applications, and Methodological Advances

arXiv:2511. 03354v2 Announce Type: replace-cross Abstract: Generative artificial intelligence (GenAI) is transforming bioinformatics by advancing genomics, proteomics, transcriptomics, structural biology, and drug discovery.

By Wasimul Karim, Riasad Alvi, Sayeem Been Zaman, Arefin Ittesafun Abian, Mohaimenul Azam Khan Raiaan, Saddam Mukta, Md Rafi Ur Rashid, Md Rafiqul Islam, Yakub Sebastian, Sami Azam

benchmarks safety

arXiv AI

Jul 24

CRAG-MM-Diagnostics: Enabling Stage-Wise Analysis of Knowledge-Intensive VQA

arXiv:2607. 21155v1 Announce Type: cross Abstract: Knowledge-Intensive Visual Question Answering (KI-VQA) benchmarks evaluate Vision-Language Models (VLMs) as multimodal knowledge assistants by requiring external information beyond a provided image to answer questions.

By Hanseok Oh, Parishad BehnamGhader, Benno Krojer, Hyunji Lee, Paul Liang, Siva Reddy, Verna Dankers

llms rag nlp multimodal benchmarks

Get the daily briefing