Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,750 stories · RSS feed

arXiv Machine Learning

Jul 7

Target-Aware Interaction-Guided Reinforcement Learning for Black-Box Node Injection Attacks on Graph Neural Networks

arXiv:2607. 04091v1 Announce Type: new Abstract: Graph Neural Networks (GNNs) have achieved remarkable performance in graph representation learning, yet their inherent vulnerability to adversarial attacks poses severe security risks.

By Yi Lan, Ye Yuan

reinforcement-learning benchmarks safety

arXiv AI

Jul 7

A Failure-Mode Benchmark for Polymorphic Sybil Poisoning in RAG

arXiv:2607. 03739v1 Announce Type: cross Abstract: We release a benchmark and failure-mode-aware evaluation framework for grounded QA under coordinated retrieval poisoning.

By Donghyun Lee (Dongguk University), Juntae Kim (Dongguk University)

rag multimodal benchmarks

arXiv AI

Jul 7

ELiTeFormer: An Efficient Transformer for FPGAs

arXiv:2607. 03652v1 Announce Type: cross Abstract: Transformer blocks are prevalent in large language model (LLM) but present deployment challenges due to their challenging computational and memory demands.

By Victor Agostinelli, Nicolas Bohm Agostini, Antonino Tumeo

llms efficiency benchmarks

arXiv AI

Jul 7

STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training

arXiv:2607. 04963v1 Announce Type: new Abstract: Reinforcement Learning (RL) is the dominant paradigm for training Large Language Model (LLM) agents on long-horizon tasks.

By Qiuyi Qi, Tian Liang, Mutian Bao, Jinjian Zhang, Dongnan Liu, Wei Zhou, Linjian Mo, Ming Kong, Jie Liu, Feng Zhang, Qiang Zhu

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 7

Toward Trustworthy Large Language Model Agents in Healthcare

arXiv:2607. 05055v1 Announce Type: new Abstract: Healthcare appointment scheduling remains a persistent operational bottleneck, driven by manual coordination, fragmented legacy systems, and high administrative overhead.

By Hadi Hasan, Safaa Salman, Adam Tai Abou Dargham, Ammar Mohanna, Ali Chehab

llms rag agents benchmarks safety

arXiv AI

Jul 7

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

arXiv:2607. 04163v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable progress in visual understanding tasks such as image captioning and visual question answering.

By Kai Tang, Jinhao You, Bohua Zhang, Yichen Guo, Yiding Sun, Dongxu Zhang, Chenxi Li, Xiande Huang, Shanghang Zhang

llms nlp multimodal benchmarks safety

arXiv Machine Learning

Jul 7

Piecewise Dynamic Diffusion Regularization for Reconstruction of Cardiac Cine MRI

arXiv:2607. 03299v1 Announce Type: cross Abstract: Real-time cardiac cine MRI enables visualization of the beating heart during free breathing, but severe undersampling and motion make reconstruction highly challenging.

By Florian F\"urnrohr, Reinhard Heckel

diffusion benchmarks

arXiv AI

Jul 7

ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs

arXiv:2601. 07475v2 Announce Type: replace-cross Abstract: The emergence of fine-grained numerical formats like NVFP4 presents new opportunities for efficient Large Language Model (LLM) inference.

By Haoqian Meng, Yilun Luo, Yafei Zhao, Wenyuan Liu, Peng Zhang, Xindian Ma

llms efficiency benchmarks

arXiv AI

Jul 7

HCSU: A Dataset and Benchmark for Fine-Grained Historical Calligraphy Style Understanding

arXiv:2607. 04147v1 Announce Type: cross Abstract: Automated fine-grained perception of calligraphy styles--a task vital to cultural heritage preservation--remains a critical challenge for Large Vision-Language Models (LVLMs), largely constrained by existing datasets that suffer from modal mixture and flattened labels.

By Yinsheng Yao, Yan Liu, Chen Ye

llms multimodal benchmarks

arXiv Machine Learning

Jul 7

TT-Sparse: Learning Sparse Rule Models with Differentiable Truth Tables

arXiv:2603. 07606v2 Announce Type: replace Abstract: Interpretable machine learning is essential in high-stakes domains where decision-making requires accountability, transparency, and trust.

By Hans Farrell Soegeng, Sarthak Ketanbhai Modi, Thomas Peyrin

benchmarks safety

arXiv AI

Jul 7

Attention Dynamics in Diffusion Models: A Visual Analytics Framework for Human-AI Collaboration

arXiv:2607. 02563v1 Announce Type: cross Abstract: Diffusion-based text-to-image models can synthesize complex and highly structured visual content, yet the emergence and evolution of semantic structure remain difficult to interpret.

By Yiran Xiao, George Legrady

diffusion benchmarks

arXiv AI

Jul 7

A Fair Benchmarking of Deep Relational Database Learning Models

arXiv:2607. 03659v1 Announce Type: cross Abstract: Relational databases (RDBs) are the primary data infrastructure in many enterprises, yet recent deep learning methods designed for RDBs have been evaluated under inconsistent experimental protocols, making fair comparison difficult.

By Kazi F. Akhter, Bharath Ajendla, Manar D. Samad

llms benchmarks

arXiv Machine Learning

Jul 7

Evaluating Time Series Foundation Models for Electricity Price Forecasting: Contamination Risk, Distributional Shifts, and Covariate Dependence

arXiv:2607. 02623v1 Announce Type: new Abstract: Time series foundation models (TSFMs) have shown strong zero-shot forecasting performance, but their generalization in covariate-driven, non-stationary settings is underexplored.

By Zhenghua Pan, Ahmed Aziz Ezzat

benchmarks

arXiv AI

Jul 7

CARL: Constraint-Aware Reinforcement Learning for Planning with LLMs

arXiv:2607. 04854v1 Announce Type: new Abstract: Despite their strong reasoning capabilities and extensive world knowledge, Large Language Models (LLMs) frequently generate plans that violate task constraints, undermining their reliability in real-world applications.

By Qiuyi Qi, Jinjian Zhang, Mutian Bao, Tian Liang, Guocong Li, Dongnan Liu, Wei Zhou, Jie Liu, Ming Kong, Linjian Mo, Feng Zhang, Qiang Zhu

llms reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 7

TSP with Predictions: Heatmap to Tour with Provable Guarantees

arXiv:2607. 03791v1 Announce Type: cross Abstract: The Traveling Salesperson Problem (TSP) has long served as a benchmark for evaluating the strength of optimization techniques in the classical theory of algorithms.

By Marek Eli\'a\v{s}, Fabrizio Grandoni, Adam Polak, Eleonora Vercesi

benchmarks

arXiv AI

Jul 7

FM-ChangeNet: Learning Change through Pathwise Feature Transport

arXiv:2607. 04750v1 Announce Type: new Abstract: We present FM-ChangeNet, a pathwise-supervised framework for change detection that reformulates bi-temporal reasoning as continuous transport in feature space rather than static endpoint comparison.

By Roie Kazoom, George Leifman, Genady Beryozkin

computer-vision benchmarks safety

arXiv AI

Jul 7

Agent vs. Parametric World Models: Hybrid Planning for Reliable Language Agents

arXiv:2606. 27806v3 Announce Type: replace Abstract: Language agents plan by generating not only actions but also implicit predictions of how the world will change.

By Xinyuan Song, Zekun Cai

llms agents benchmarks

arXiv AI

Jul 7

Decentralized Aggregation of LLM Predictions via Wagering Mechanisms

arXiv:2607. 04389v1 Announce Type: new Abstract: It is increasingly common to aggregate predictions from multiple LLMs, each with domain expertise or access to private tools and data, to improve collective prediction performance.

By Yuhong Luo, David M. Pennock, Xintong Wang

llms benchmarks safety

arXiv AI

Jul 7

DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collections

arXiv:2508. 12116v2 Announce Type: replace-cross Abstract: As numerous instruction-tuning datasets continue to emerge, dynamically balancing and optimizing their mixtures has become a critical challenge.

By Haebin Shin, Lei Ji, Xiao Liu, Zhiwei Yu, Hyunwoo Yoo, Qi Chen, Yeyun Gong

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 7

CanniUplift: A Holistic Framework for Mitigating Seller and Incentive Cannibalization in E-commerce Uplift Modeling

arXiv:2607. 05242v1 Announce Type: cross Abstract: Personalized incentive allocation is vital for e-commerce, where uplift modeling is the standard for estimating Individual Treatment Effects (ITE).

By Zuwang He, Shihao Shu, Yuli Qu, Hanyu Gao, Ziliang Zhang, Diwei Chen, Xiangda Yan, Buyu Gao, Tanchao Zhu, Yumeng Li, Junxiong Zhu

benchmarks safety

Get the daily briefing