Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,930 stories · RSS feed

arXiv AI

Jul 1

RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora

arXiv:2604. 19047v2 Announce Type: replace-cross Abstract: Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate on corpora such as financial reports, legal codes, and patents, where information is highly redundant and documents exhibit strong inter-document similarity.

By Hanjun Cho, Jay-Yoon Lee

llms rag benchmarks

arXiv AI

Jul 1

On the Convergence of Self-Improving Online LLM Alignment

arXiv:2606. 31524v1 Announce Type: cross Abstract: The Self-Improving Alignment (SAIL) algorithm addresses distribution shift by reducing a bilevel formulation of the problem to an efficient, single-level method.

By Xudong Wu, Pangpang Liu, Vaneet Aggarwal, Jiayu Chen

llms benchmarks safety

arXiv AI

Jul 1

PruneGround: Plug-and-play Spatial Pruning for 3D Visual Grounding

arXiv:2606. 31148v1 Announce Type: cross Abstract: 3D Visual Grounding (3DVG) aims to localize target objects in 3D scenes given natural language descriptions.

By Duc Cao Dinh, Khai Le-Duc, Florent Draye, Chris Ngo, Terry Jingchen Zhang, Bernhard Sch\"olkopf, Zhijing Jin

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jul 1

L-SR1: Learned Symmetric-Rank-One Preconditioning

arXiv:2508. 12270v3 Announce Type: replace Abstract: End-to-end deep learning has achieved impressive results but often relies on large labeled datasets, exhibits limited generalization to unseen scenarios, and incurs substantial computational cost.

By Gal Lifshitz, Shahar Zuler, Ori Fouks, Dan Raviv

fine-tuning benchmarks

arXiv AI

Jul 1

Gated Multi-Graph Fusion via Graph Attention Networks for Alzheimer's Disease Detection

arXiv:2606. 31186v1 Announce Type: cross Abstract: Spontaneous speech is a vital non-invasive biomarker for Alzheimer's Disease (AD), yet many systems overlook non-linear structural disruptions and clinical heterogeneity in pathological language.

By Jinyu Li, Xiao Wei, Bin Wen, Kai Li, Yuqin Lin, Xiaobao Wang, Longbiao Wang, Jianwu Dang

multimodal benchmarks

arXiv AI

Jul 1

ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences

arXiv:2602. 11354v3 Announce Type: replace Abstract: The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers.

By Bang Nguyen, Dominik So\'os, Qian Ma, Rochana R. Obadage, Zack Ranjan, Sai Koneru, Timothy M. Errington, Shakhlo Nematova, Sarah Rajtmajer, Jian Wu, Meng Jiang

llms agents benchmarks

arXiv Machine Learning

Jul 1

Relational and Sequential Conformal Inference for Energy Time Series over Graphs via Foundation Models

arXiv:2606. 31804v1 Announce Type: new Abstract: Accurate energy demand forecasting is essential for the reliable operation and planning of modern sustainable energy systems.

By Keivan Faghih Niresi, Alice Cicirello, Olga Fink

llms benchmarks safety

arXiv Machine Learning

Jul 1

Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability

arXiv:2601. 18778v3 Announce Type: replace Abstract: RL methods for scaling large reasoning models stall on datasets with low initial success rates, and thus little training signal.

By Shobhita Sundaram, John Quan, Ariel Kwiatkowski, Kartik Ahuja, Yann Ollivier, Julia Kempe

llms reinforcement-learning benchmarks

arXiv AI

Jul 1

ALM2Vec: Learning Audio Embeddings for Universal Audio Retrieval with Large Audio-Language Models

arXiv:2606. 30682v1 Announce Type: cross Abstract: Recent advances in language--audio retrieval have been largely driven by contrastive dual-encoder architectures that align audio and text in a shared embedding space.

By Fengjie Lu, Chenang Jiang, Jiarui Hai, Helin Wang, Aaron Yee

llms rag nlp multimodal benchmarks

arXiv Machine Learning

Jul 1

Targeted Tests for LLM Reasoning: An Audit-Constrained Protocol

arXiv:2605. 11599v3 Announce Type: replace Abstract: Fixed reasoning benchmarks evaluate canonical prompts, but semantically valid changes in presentation can still change model behavior.

By Hongmin Li

llms benchmarks

arXiv Machine Learning

Jul 1

Sparse POD Mode Selection and Manifold Dimensionality Reduction with Neural Networks

arXiv:2605. 27756v2 Announce Type: replace-cross Abstract: Linear dimensionality reduction methods such as proper orthogonal decomposition (POD) make high-dimensional data amenable to analysis by identifying the principal components, or modes, that capture the most variance, or energy, in the data and constructing a low-dimensional representation in the subspace they span.

By Tomoki Koike, Prakash Mohan, Marc T. Henry de Frahan, Elizabeth Qian, Julie Bessac

efficiency benchmarks safety

arXiv AI

Jul 1

Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models

arXiv:2606. 31846v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models offer a promising framework for robotic manipulation by connecting language instructions, visual observations, and continuous control.

By Lang Cao, Renhong Chen, Luyi Li, Peng Wang, Mofan Peng, Yitong Li

reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv AI

Jul 1

Mantis: Lightweight Foundation Model for Time Series Classification

arXiv:2502. 15637v2 Announce Type: replace-cross Abstract: While foundation models have revolutionized various domains, their application to time series classification remains rather under-explored, with existing literature predominantly focused on forecasting.

By Vasilii Feofanov, Songkang Wen, Shifeng Xie, Simon Roschmann, Marius Alonso, Hongbo Guo, Romain Ilbert, Malik Tiomoko, Quentin Bouniot, Zeynep Akata, Lujia Pan, Jianfeng Zhang, Ievgen Redko

llms rag nlp benchmarks

arXiv AI

Jul 1

Dataset Construction for Training LLM to Learn Analog Circuit Knowledge

arXiv:2508. 10409v3 Announce Type: replace-cross Abstract: This paper constructs a textual dataset for training large language models (LLMs) to learn analog circuit knowledge and customizes LLM training techniques.

By Zihao Chen, Ji Zhuang, Jinyi Shen, Xiaoyue Ke, Xinyi Yang, Mingjie Zhou, Zhuoyao Du, Xu Yan, Zhouyang Wu, Zhenyu Xu, Jiangli Huang, Li Shang, Xuan Zeng, Fan Yang

llms agents fine-tuning benchmarks

arXiv AI

Jul 1

SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA

arXiv:2504. 07385v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness.

By Sher Badshah, Ali Emami, Hassan Sajjad

llms agents benchmarks

arXiv Machine Learning

Jul 1

The Calibration Turn in AI-Assisted Research: A Conceptual and Methodological Framework for Evidence-Licensed Claims

arXiv:2606. 31273v1 Announce Type: new Abstract: AI-assisted research has entered a stage in which the central question is not only whether systems can generate hypotheses, run experiments, or produce manuscripts, but whether their scientific claims are calibrated to the evidence that supports them.

By Hongmin Li

llms agents robotics benchmarks

arXiv AI

Jul 1

Measuring & Mitigating Over-Alignment for LLMs in Multilingual Criminal Law Courts

arXiv:2606. 23375v2 Announce Type: replace-cross Abstract: While the wider applicability of LLMs in the legal field is currently debated due to their reliability and the gravity of any errors, narrow uses with well-understood and mitigated risks have emerged.

By Arthur Wuhrmann, Gaetan Stein, Daniel Brunner, Andrei Kucharavy

llms nlp benchmarks safety

arXiv Machine Learning

Jul 1

Calibration, Not Compilation: Detecting and Repairing Misspecified Probabilistic Programs Written by Language Models

arXiv:2606. 31630v1 Announce Type: new Abstract: Language models increasingly write probabilistic programs (in NumPyro, Stan, or Pyro), but a program that compiles, runs, and passes every unit test can still be \emph{statistically} wrong -- a Gaussian likelihood for heavy-tailed data, a Poisson for over-dispersed counts, an invalid prior support, or a pathological parameterization.

By Jian Xu, Delu Zeng, John Paisley, Qibin Zhao

llms benchmarks

arXiv Machine Learning

Jul 1

Interface-Aware Neural Newton Preconditioning for Robust Cohesive Zone Model Simulations

arXiv:2606. 31921v1 Announce Type: new Abstract: Cohesive Zone Models (CZMs) are widely used to simulate interface fracture, delamination, adhesive failure, and fiber--matrix debonding in aerospace composite structures.

By Zhangyong Liang, Huanhuan Gao

benchmarks

arXiv Machine Learning

Jul 1

Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks

arXiv:2606. 31700v1 Announce Type: new Abstract: Biological neural circuits obey Dale's principle: each neuron's synapses are uniformly excitatory or inhibitory.

By Yutaro Yamada, Luca Grillotti, Rujikorn Charakorn, Sebastian Risi, David Ha, Robert Tjarko Lange

diffusion reinforcement-learning benchmarks safety

Get the daily briefing