Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,077 stories · RSS feed

arXiv AI

Jul 21

LaT: LLM-as-Trainer for Multi-Task Vehicle Routing Solvers

arXiv:2607. 17708v1 Announce Type: new Abstract: Multi-task neural solvers aim to handle multiple Vehicle Routing Problem (VRP) variants within a unified model, avoiding separate training for each constraint combination.

By Yang Wang, Ya-Hui Jia, Wei-Neng Chen, Yi Mei, Wen Song, Zhiguang Cao

llms benchmarks

arXiv AI

Jul 21

SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

arXiv:2607. 16252v1 Announce Type: cross Abstract: Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.

By Yupeng Chang, Yuan Wu, Yi Chang

llms fine-tuning benchmarks

arXiv AI

Jul 21

LenGuard-GPC: Length Guarding with Guided-Prompt Consistency for Spatial Reasoning Reinforce Learning

arXiv:2607. 17243v1 Announce Type: new Abstract: Multi-view spatial reasoning requires vision-language models to compare visual evidence across images, align object correspondences, and infer spatial relations over long visual contexts, a setting where chain-of-thought reasoning tends to grow verbose without becoming more accurate.

By Xingjian Tao, Yiwei Wang, Yujun Cai, Jing Tang

llms reinforcement-learning multimodal benchmarks

arXiv AI

Jul 21

Reducing Per-Sample Harm in Stochastic Optimization

arXiv:2607. 16261v1 Announce Type: cross Abstract: Modern optimizers combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments.

By Apostolos Avranas

computer-vision benchmarks

arXiv Machine Learning

Jul 21

LFM: Leveraging Foundation Models for Source-Free Universal Domain Adaptation

arXiv:2607. 17653v1 Announce Type: cross Abstract: Source-free universal domain adaptation (SF-UniDA) adapts a pre-trained source model to an unlabeled target domain under both covariate and label shifts, without access to source data.

By Jing Li, Pan Liu, Meng Zhao, Wanli Xue, Yanhong Yang, Xu Cheng, Fan Shi, Jianhua Zhang, Qinghua Hu, Shengyong Chen

llms fine-tuning multimodal benchmarks

arXiv AI

Jul 21

Benchmarking Machine Learning Models for Multi-Omics-Based Breast Cancer Prediction

arXiv:2607. 16250v1 Announce Type: cross Abstract: Estrogen Receptor (ER) status is a critical biomarker in breast cancer diagnosis, prognosis, and treatment selection.

By Priyanka Paudel, Madan Baduwal

efficiency benchmarks

arXiv Machine Learning

Jul 21

Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimization

arXiv:2601. 13359v3 Announce Type: replace-cross Abstract: Prefill attacks are an effective and low-cost jailbreaking method, as they directly insert an acceptance sequence (e.

By Asen Dotsinski, Panagiotis Eustratiadis

llms multimodal benchmarks safety

arXiv AI

Jul 21

HAS: Highlight-guided Attention Steering for Multimodal LLM Video Summarization

arXiv:2607. 17994v1 Announce Type: cross Abstract: Video understanding has become more and more important with the growth of Artificial Intelligence (AI) for video generation.

By Rui Chu, Yingjie Lao

llms diffusion nlp multimodal benchmarks

arXiv AI

Jul 21

SAGA: Synthetic Agentic Graph Architecture for Temporal Benchmark Generation

arXiv:2607. 17288v1 Announce Type: cross Abstract: High quality temporal graph benchmarks with rich semantics and ground-truth anomaly labels are essential for training graph neural networks, yet remain scarce due to privacy constraints and annotation costs.

By Jiacheng Ding, Xiaofei Zhang

llms rag agents benchmarks safety

arXiv AI

Jul 21

Certified-Gap Dual-Price Policies for Real-Time Truckload Bid Acceptance with Relocating, Clock-Constrained Resources

arXiv:2607. 16891v1 Announce Type: cross Abstract: A truckload carrier must accept or reject each load tender within seconds.

By Aswin Chandrasekaran

benchmarks

arXiv Machine Learning

Jul 21

Adaptive Mamba Neural Operators

arXiv:2607. 18043v1 Announce Type: new Abstract: Accurately solving partial differential equations (PDEs) on arbitrary geometries and a variety of meshes is an important task in science and engineering applications.

By Zeyuan Song, Zheyu Jiang

benchmarks

arXiv Machine Learning

Jul 21

Program Synthesis for Simulation-Based Inference: Joint Model Selection and Parameter Estimation

arXiv:2607. 17540v1 Announce Type: new Abstract: Neural simulation-based inference enables parameter estimation for complex models, but typically requires the user to specify a simulator encoding a fixed model structure.

By Siddharth Mishra-Sharma

llms benchmarks

arXiv AI

Jul 21

WuYu-EnvLE-Bench: A Benchmark for Evaluating Large Language Models in Environmental Law Enforcement

arXiv:2607. 17745v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly considered for environmental enforcement, but their ability to produce traceable enforcement decisions remains unclear.

By Ziliang Yang, Yi Zhang, Kaijun Lin, Jiachao Ke, Haihong Xu, Zongguo Wen

llms benchmarks

arXiv AI

Jul 21

After the Euclidean Highway: Hyperbolic Expert AI as the Next Innovation

arXiv:2607. 17513v1 Announce Type: cross Abstract: Expert domains are trees; the Euclidean transformer is not, diluting parent-child structure exponentially at depth.

By Kwan Soo Shin, In Seok Kang, Munho Lee

llms benchmarks

arXiv AI

Jul 21

Tractable Query Answering under Epistemic Confidentiality Policies in DL Ontologies (extended version)

arXiv:2607. 16715v1 Announce Type: new Abstract: We study Controlled Query Evaluation (CQE), a declarative approach to confidentiality-preserving data access, in the context of Description Logic (DL) ontologies, and for confidentiality policies expressed through Epistemic Dependencies (EDs).

By Lorenzo Marconi, Daniela Rieti, Riccardo ROsati

benchmarks

arXiv AI

Jul 21

A Predict-then-Correct Loop Based on Few-Shot Continuous Contextual Bandit for Demand Forecasting

arXiv:2607. 16354v1 Announce Type: cross Abstract: Retail demand forecasting remains difficult when demand shifts faster than static forecasting models can be retrained, especially in early demand cycles where newly observed labels are sparse.

By Zhiwei Lei, Benedict Jun Ma, Ilya Jackson

reinforcement-learning benchmarks

arXiv AI

Jul 21

Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning

arXiv:2505. 20161v2 Announce Type: replace-cross Abstract: Effective generalization in language models depends critically on the diversity of their training data.

By Jaehun Jung, Seungju Han, Ximing Lu, Skyler Hallinan, David Acuna, Shrimai Prabhumoye, Mostafa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Yejin Choi

llms benchmarks

arXiv AI

Jul 21

BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis

arXiv:2607. 17782v1 Announce Type: cross Abstract: Foundation models pretrained using self-supervised learning have transformed computer vision by learning transferable representations from large-scale unlabeled data.

By Moona Mazher, Abdul Qayyum, Steven A. Niederer, Daniel C. Alexander

computer-vision fine-tuning benchmarks

arXiv AI

Jul 21

Scalable Causal Imitation Learning

arXiv:2607. 17003v1 Announce Type: cross Abstract: Imitation learning enables learning a policy in an unknown environment with a latent reward signal using expert demonstrations, but it struggles when the imitator's and expert's observations are mismatched and unobserved confounders are present in expert demonstrations.

By Eylam Tagor, Mingxuan Li, Elias Bareinboim

reinforcement-learning benchmarks safety

arXiv AI

Jul 21

A Geometric Perspective on Stabilizing Value Conflict Resolution

arXiv:2607. 17946v1 Announce Type: cross Abstract: Large Language Models (LLMs) often struggle to navigate value conflicts when trained with the compressed scalar rewards of Reinforcement Learning from Human Feedback (RLHF).

By Saket Reddy, Andy Liu

llms reinforcement-learning benchmarks safety

Get the daily briefing