Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,941 stories · RSS feed

arXiv AI

Jun 8

Proxy Reconstruction Pre-training for Ramp Flow Prediction at Highway Interchanges

arXiv:2510. 03381v3 Announce Type: replace-cross Abstract: Interchanges are crucial nodes for vehicle transfers between highways, yet the lack of real-time ramp detectors creates blind spots in traffic prediction.

By Yongchao Li, Jun Chen, Zhuoxuan Li, Chao Gao, Yang Li, Chu Zhang, Changyin Dong

multimodal benchmarks

arXiv AI

Jun 8

HKJudge: A Legal Discourse-Annotated Corpus for Interpreting What Courts Find, How They Reason, and What They Rule

arXiv:2606. 06679v1 Announce Type: cross Abstract: Court judgments are central to legal practice and jurisprudence, yet discourse analysis of Hong Kong judgments has received limited attention, owing largely to the absence of expert-annotated corpora.

By Xi Xuan, Wenxin Zhang, Yufei Zhou, King-kui Sin, Chunyu Kit

llms fine-tuning benchmarks

arXiv AI

Jun 8

LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis

arXiv:2510. 24561v3 Announce Type: replace-cross Abstract: LoRA has become a widely adopted method for PEFT, and its initialization methods have attracted increasing attention.

By Qingyue Zhang, Chang Chu, Tianren Peng, Qi Li, Xiangyang Luo, Zhihao Jiang, Shao-Lun Huang

fine-tuning benchmarks safety

arXiv AI

Jun 8

DuMate-DeepResearch: An Auditable Multi-Agent System with Recursive Search and Rubric-Grounded Reasoning

arXiv:2606. 07299v1 Announce Type: new Abstract: Deep Research (DR) has emerged as a new agentic paradigm to tackle complex, open-ended research tasks, demanding systems that can iteratively frame problems, acquire evidence, verify sources, and synthesize long-form reports.

By Lingyong Yan, Can Xu, Yukun Zhao, Wenxuan Li, Qingyang Chen, Jiulong Wu, Wenli Song, Xiangnan Li, Weixian Shi, Yiqun Chen, Xuchen Ma, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Jianmin Wu, Dawei Yin

agents benchmarks safety

arXiv Machine Learning

Jun 8

Learnable Kernel Density Estimation for Graphs and Its Application to Graph-Level Anomaly Detection

arXiv:2505. 21285v5 Announce Type: replace Abstract: This work proposes a framework LGKDE that learns kernel density estimation for graphs.

By Xudong Wang, Ziheng Sun, Chris Ding, Jicong Fan

benchmarks

arXiv AI

Jun 8

MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

arXiv:2605. 05225v3 Announce Type: replace-cross Abstract: Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect.

By Bo Li, Chuan Wu, Shaolin Zhu

llms multimodal benchmarks

arXiv AI

Jun 8

Which Anatomy Matters Under Limited Labels? A Data-Efficient Anatomy-Aware Benchmark for Cardiac Pathology Prediction

arXiv:2606. 06509v1 Announce Type: cross Abstract: Numerous medical imaging problems must be solved under limited labels and constrained compute, yet it remains unclear whether performance gains are driven mainly by more expressive models or by better representation of clinically meaningful anatomy.

By Himanshu Singh

computer-vision benchmarks

arXiv Machine Learning

Jun 8

Breaking the Ice: Analyzing Cold Start Latency in vLLM

arXiv:2606. 07362v1 Announce Type: new Abstract: As scalable inference services become popular, the cold start latency of an inference engine becomes important.

By Huzaifa Shaaban Kabakibo, Animesh Trivedi, Lin Wang

benchmarks

arXiv Machine Learning

Jun 8

Bradley-Terry Rankings for Recommender Systems Across Dataset Taxonomies

arXiv:2606. 07492v1 Announce Type: cross Abstract: The ranking of recommendation algorithms is a challenging problem since model performance is sensitive to dataset characteristics such as sparsity, sequential structure, and scale.

By Ekaterina Grishina, Stepan Kuznetsov, Askar Tsyganov, Ilya Ivanov, Daria Korovaitceva, Margarita Rusanova, Uliana Parkina, Alexander Derevyagin, Evgeny Frolov, Sergey Samsonov, Anton Lysenko

efficiency benchmarks

arXiv AI

Jun 8

Synthetic Benchmarks Overstate Forward-Forward Scaling: Real-Data Limits of Layer-Local Training

arXiv:2606. 06539v1 Announce Type: cross Abstract: Forward-Forward (FF) learning [Hinton, 2022] replaces backpropagation with strictly layer-local goodness updates.

By Yucheng Chen

benchmarks

arXiv AI

Jun 8

Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models

arXiv:2606. 06534v1 Announce Type: cross Abstract: Longitudinal medical visual question answering (VQA) requires reasoning about anatomical differences between an image of a current time point and an image of a referred time point.

By Jialin Wu, Qianru Zhang, Georges El Fakhri, Xiaofeng Liu

llms nlp multimodal benchmarks safety

arXiv AI

Jun 8

Think Like a Pilot: Fine-Grained Long-Horizon UAV Navigation

arXiv:2606. 06836v1 Announce Type: cross Abstract: Language-guided UAV agents must execute long-horizon semantic instructions while producing smooth, physically feasible continuous flight commands, yet existing Vision-Language Navigation (VLN) benchmarks typically use discrete or coarse actions and existing UAV Vision-Language-Action (VLA) tasks focus on short, atomic maneuvers.

By Xiangyi Zheng, Xiangyu Wang, Qinan Liao, Zimu Tang, Yue Liao, Dongyue Lyu, Guodong Wang, Junjie Liu, Si Liu

llms agents diffusion multimodal benchmarks

arXiv Machine Learning

Jun 8

On the Effect of Neural Field Reparameterization for 4DVAR

arXiv:2509. 21751v2 Announce Type: replace Abstract: Four-dimensional variational data assimilation (4DVAR) is a cornerstone of numerical weather prediction, yet it remains computationally intensive and sensitive to initialization due to the non-convexity of its objective function.

By Jaemin Oh

benchmarks safety

arXiv Machine Learning

Jun 8

Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returns

arXiv:2503. 03660v4 Announce Type: replace Abstract: We introduce a sequence-conditioned critic for Soft Actor-Critic (SAC) that models trajectory context with a lightweight Transformer and trains on aggregated $N$-step targets.

By Dong Tian, Onur Celik, Gerhard Neumann

llms rag reinforcement-learning benchmarks

arXiv AI

Jun 8

Extracting Recurring Vulnerabilities from Black-Box LLM-Generated Software

arXiv:2602. 04894v4 Announce Type: replace-cross Abstract: LLMs are increasingly used for code generation, but their outputs often follow recurring templates that can induce predictable vulnerabilities.

By Tomer Kordonsky, Amit LeVi, Maayan Yamin, Noam Benzimra, Avi Mendelson

llms benchmarks

arXiv AI

Jun 8

CULTURESCORE: Evaluating Cultural Faithfulness in Video Generation Models

arXiv:2606. 07311v1 Announce Type: cross Abstract: As video generation models like Veo 3.

By Anku Rani, Wei Dai, Shravan Nayak, Pattie Maes, Mahdi M. Kalayeh, Paul Pu Liang

diffusion benchmarks

arXiv Machine Learning

Jun 8

TALAN: Task-Aligned Latent Adaptation Networks for Targeted Post-Training of Large Language Models

arXiv:2606. 06902v1 Announce Type: new Abstract: Targeted post-training aims to improve reasoning, math, and code without degrading strengths.

By Chengkai Zhang, Ziteng Liu, Junpu Wang, Zeyi Tao, Yang Wang, Sagar Chordia, Qin Huang

llms fine-tuning benchmarks

arXiv AI

Jun 8

It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agents

arXiv:2512. 23128v2 Announce Type: replace-cross Abstract: Web-based agents powered by large language models are increasingly used for tasks such as email management or professional networking.

By Karolina Korgul, Yushi Yang, Arkadiusz Drohomirecki, Piotr B{\l}aszczyk, Will Howard, Lukas Aichberger, Chris Russell, Philip H. S. Torr, Adam Mahdi, Adel Bibi

llms agents benchmarks safety

arXiv AI

Jun 8

FAIR-Calib: Frontier-Aware Instability-Reweighted Calibration for Post-Training Quantization of Diffusion Large Language Models

arXiv:2606. 06547v1 Announce Type: cross Abstract: Diffusion Large Language Models (dLLMs) refine tokens iteratively but commit them irreversibly, leading to a "stability lag" where early decisions remain fragile even after being written.

By Haoyu Huang, Linlin Yang, Sheng Xu, Boyu Liu, Guodong Guo, Zhongqian Fu, Hang Zhou, Baochang Zhang

llms diffusion efficiency benchmarks

arXiv AI

Jun 8

ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios

arXiv:2602. 16073v2 Announce Type: replace-cross Abstract: Developing autonomous driving systems for complex traffic environments requires balancing multiple objectives, such as avoiding collisions, obeying traffic rules, and making efficient progress.

By Kevin Kai-Chun Chang, Ekin Beyazit, Alberto Sangiovanni-Vincentelli, Tichakorn Wongpiromsarn, Sanjit A. Seshia

agents robotics benchmarks

Get the daily briefing