Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,794 stories · RSS feed

arXiv AI

Jul 31

ARC-Encoder: learning compressed text representations for large language models

arXiv:2510. 20535v2 Announce Type: replace-cross Abstract: Recent techniques such as retrieval-augmented generation or chain-of-thought reasoning have led to longer contexts and increased inference costs.

By Hippolyte Pilchen, Edouard Grave, Patrick P\'erez

llms rag fine-tuning benchmarks

arXiv AI

Jul 31

DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English

arXiv:2601. 22888v4 Announce Type: replace-cross Abstract: More than 80% of the 1.

By Jio Oh, Paul Vicinanza, Thomas Butler, Steven Euijong Whang, Dezhi Hong, Amani Namboori

llms benchmarks data-engineering

arXiv AI

Jul 31

PatchDenoiser: Parameter-efficient multi-scale patch learning and fusion denoiser for Low-dose CT imaging

arXiv:2602. 21987v3 Announce Type: replace-cross Abstract: Low-dose CT images are essential for reducing radiation exposure in cancer screening, pediatric imaging, and longitudinal monitoring protocols, but their quality is often degraded by noise from low-dose acquisition, patient motion, or scanner limitations, affecting both clinical interpretation and downstream analysis.

By Jitindra Fartiyal, Pedro Freire, Sergei K. Turitsyn, Sergei G. Solovski

llms diffusion fine-tuning benchmarks

arXiv AI

Jul 31

Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge

arXiv:2603. 07131v4 Announce Type: replace-cross Abstract: Large Vision Language Models (LVLMs) show immense potential for automated ophthalmic diagnosis.

By Shuai Lu, Meng Wang, Jia Guo, Jiawei Du, Bo Liu, Shengzhu Yang, Weihang Zhang, Huazhu Fu, Huiqi Li

llms rag nlp benchmarks safety

arXiv AI

Jul 31

The Fast Lane Hypothesis: Von Economo Neurons Implement a Biological Speed-Accuracy Tradeoff

arXiv:2604. 09229v2 Announce Type: replace-cross Abstract: von Economo neurons (VENs) are large bipolar projection neurons found exclusively in the anterior cingulate cortex (ACC) and frontal insula of species with complex social cognition, including humans, great apes, cetaceans, and elephants.

By Esila Keskin

benchmarks

arXiv AI

Jul 31

VistaHop: Benchmarking Long-Horizon Visual DeepSearch

arXiv:2606. 03273v2 Announce Type: replace-cross Abstract: Visual DeepSearch tasks require multimodal large language models (MLLMs) to resolve complex visual queries by repeatedly inspecting image regions, grounding reasoning in visual evidence, and connecting fine-grained clues across multiple steps.

By Hang He, Chuhuai Yue, Chengqi Dong, Chengcheng Wan, Ting Su, Haiying Sun, Jiajun Chai, Xiaohan Wang, Guojun Yin

llms agents multimodal benchmarks

arXiv AI

Jul 31

Adversarial Pragmatics for AI Safety Evaluation: A Diagnostic Framework and Seed Benchmark for Language-Mediated Control

arXiv:2607. 01153v3 Announce Type: replace-cross Abstract: Safety evaluations for language models increasingly depend on judgments about ambiguous natural-language behaviour: whether a model followed an instruction, refused appropriately, complied with a policy, or misreported progress in an agentic task.

By Brett Reynolds

llms agents benchmarks safety

arXiv Machine Learning

Jul 31

PlatformBid: An Auto-Bidding Benchmark from a Unified Advertising Platform's Perspective

arXiv:2607. 27265v1 Announce Type: new Abstract: Real-time bidding is central to computational advertising, comprising three elements: Supply Side Platform (SSP) selling ad impressions, Demand Side Platform (DSP) bidding for advertisers, and Ad Exchange conducting auctions between them.

By Shengtian Yang, Yewen Li, Peng Jiang, Zhiyi Lyu, Bo An, Peng Jiang, Qingpeng Cai, Lei Feng

reinforcement-learning benchmarks

arXiv Machine Learning

Jul 31

DoTime: A Synthetic Benchmark Generator for Interventional and Counterfactual Time Series

arXiv:2607. 27263v1 Announce Type: new Abstract: Most benchmarks for causal inference over time series are observational, small, or domain-specific, leaving interventional and counterfactual estimation under-served exactly where it matters most, such as in healthcare, policy evaluation, and climate science.

By Dennis Thumm, Billy Tim Anthony, Ying Chen

benchmarks

arXiv Machine Learning

Jul 31

EvoCause: LLM-Guided Evolution of Causal Graphs for Root Cause Analysis

arXiv:2607. 27290v1 Announce Type: new Abstract: Modern telecommunication, cloud, and microservice systems emit correlated alarm cascades when components fail.

By Lei Zan, Keli Zhang, Shifeng Xie, Jiale Zheng, Zehao Xiao, Zhiwei Dong, Ke Zhang, Ruichu Cai, Malik Tiomoko, Lujia Pan

llms benchmarks safety

arXiv Machine Learning

Jul 31

TIER-MoE: Trust-Informed Expert Routing via Conditional Modality Risk for Multimodal Fusion in Biomedical Classification

arXiv:2607. 27289v1 Announce Type: new Abstract: The promise of multimodal fusion lies in combining complementary sources of evidence, yet more evidence does not always yield a better prediction.

By Yu Chang, Anzhe Cheng, Chenwei Wu, Zhuoran Wang, Jiahao Chen, Tamoghna Chattopadhyay, Sophia I. Thomopoulos, Paul M. Thompson, Liyue Shen, Paul Bogdan

multimodal benchmarks

arXiv Machine Learning

Jul 31

Expanding Data-Agnostic Pivotal Instances Selection Models with Proximity Trees and Ensemble Learning

arXiv:2607. 27522v1 Announce Type: new Abstract: As decision-making processes grow more complex, machine learning tools have become essential for tackling business and societal challenges.

By Alessio Cascione, Mattia Setzu, Cristiano Landi, Paolo Maria Mancarella, Riccardo Guidotti

benchmarks

arXiv Machine Learning

Jul 31

Latent-Kernel Discrete Flow Maps for Few-Step Generation

arXiv:2607. 27529v1 Announce Type: new Abstract: Discrete diffusion and flow-matching models denoise a sequence over many steps, but to keep each step cheap, they factorize the transition across positions and decide every token independently.

By Mansoor Ahmed, Yue-Tsz Fan, Hemanth Venkateswara, Murray Patterson

llms diffusion benchmarks

arXiv Machine Learning

Jul 31

A Montage-Agnostic Encoder for Calibration-Light Cross-User Gesture Recognition from Surface Electromyography

arXiv:2607. 27565v1 Announce Type: new Abstract: Pattern-recognition control promises a myoelectric prosthesis that responds to many intended gestures rather than one or two, but the promise has stayed in the laboratory.

By Jethro Odeyemi, W. J. Zhang

benchmarks

arXiv Machine Learning

Jul 31

Back from the Future: Key-Value Cache Management by Counter-Causal Surprise

arXiv:2607. 27600v1 Announce Type: new Abstract: Key-value (KV) cache management through compression and eviction strategies has emerged as an important research direction in recent years.

By Stephen Gould, Anton van den Hengel

llms efficiency multimodal benchmarks

arXiv Machine Learning

Jul 31

Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning

arXiv:2607. 27610v1 Announce Type: new Abstract: Reinforcement learning (RL) finetuning significantly enhances the reasoning capabilities of large language models (LLMs), yet its effectiveness critically depends on selecting prompts of appropriate difficulty for the current policy.

By Haodong Zhu, Yangyang Ren, Yanjing Li, Sheng Xu, Haiguang Liu, Linlin Yang, Baochang Zhang

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 31

Tight Sample Complexity for Low-Rank Adaptation: Matching Bounds and Rank Selection

arXiv:2607. 27680v1 Announce Type: new Abstract: Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood.

By Arunan J

fine-tuning benchmarks

arXiv Machine Learning

Jul 31

Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning

arXiv:2607. 27968v1 Announce Type: new Abstract: Machine unlearning seeks to selectively remove specific knowledge from trained language models without full retraining, a growing necessity under privacy regulations such as GDPR and the EU AI Act.

By Efstratios Zaradoukas, Davide Gabrielli, Bardh Prenkaj, Gjergji Kasneci

llms reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Jul 31

Contrastive Reinforced Policy Optimization via Privileged Self-Distillation

arXiv:2607. 28026v1 Announce Type: new Abstract: Recent advances in post-training Large Language Models (LLMs) increasingly rely on Reinforcement Learning with Verifiable Rewards (RLVR) or On-Policy Self-Distillation (OPSD).

By Xingjian Wu, Junlin Liu, Xingchen Liu, Xuhang Zhu, Jianing Wang, Linsen Guo, Xiaoyu Li, Xuezhi Cao, Xunliang Cai

llms agents reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Jul 31

Enhancing Irregular Time Series Forecasting with Continuous-Time Modeling Framework

arXiv:2607. 28035v1 Announce Type: new Abstract: Irregular multivariate time series are widely encountered in applications such as healthcare monitoring, human activity recognition, and environmental sensing.

By Tianen Shen, Zhengyu Li, Yutong Li, Xiangfei Qiu, Xingjian Wu, Bin Yang, Jilin Hu

llms diffusion nlp benchmarks

Get the daily briefing