Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,829 stories · RSS feed

arXiv AI

Jul 23

Generative Augmented Inference of LLM-generated Data for Market Research: Theory and Empirical Evidence

arXiv:2604. 14575v3 Announce Type: replace-cross Abstract: Marketing research often relies on parameters estimated from costly human-generated data, such as conjoint survey responses, purchase decisions, and field experiment outcomes.

By Cheng Lu, Mengxin Wang, Dennis J. Zhang, Heng Zhang

llms benchmarks

arXiv Machine Learning

Jul 23

JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Models

arXiv:2607. 17572v2 Announce Type: replace Abstract: Group Relative Policy Optimization (GRPO) is a powerful reinforcement learning algorithm for aligning generative models with human preferences.

By Ruiyi Ding, Jie Li, He Kang, Ziyan Liu, Chengru Song, Yuan chen

llms diffusion reinforcement-learning benchmarks

arXiv AI

Jul 23

PhenSPINE: A Standardized Benchmark for Spine Pathology Diagnosis

arXiv:2607. 19696v1 Announce Type: cross Abstract: The accurate diagnosis of spinal pathologies depends heavily on radiological interpretation, yet automated systems are hindered by the lack of diverse, high-quality benchmarks.

By Duong Ngoc Vu, Hai Son Nguyen, Trong-Nghia Nguyen, Bien Tran Van, Trang Mai Xuan, Huan Vu, Thien Van Luong

benchmarks

arXiv AI

Jul 23

Geometry-Guided Constraint Learning for LLM Safety Classification

arXiv:2607. 19366v1 Announce Type: new Abstract: Safety as Polytope (SaP) learns linear half-space constraints in LLM hidden space but requires per-category tuning of the constraint count K.

By Fumiaki Uehara, Koo Imai, Masato Tsutsumi, Keigo Kansa, Sora Usui, Yuki Kobiyama

llms benchmarks safety

arXiv AI

Jul 23

OpenEvoShield: Dual Non-Stationary Continual Defense for Open-World Multi-Agent System Attacks

arXiv:2607. 19351v1 Announce Type: new Abstract: LLM-based multi-agent systems (LLM-MAS) are increasingly deployed in safety-critical applications, where adversaries inject malicious instructions through inter-agent communication to propagate harmful behaviors.

By Litian Zhang, Chaozhuo Li, Yuting Zhang, Zejian Chen, Bingyu Yan, Qiwei Ye

llms agents benchmarks safety

arXiv Machine Learning

Jul 23

CruiseBench: A Real-Flight-Aligned N-CMAPSS Benchmark for Engine RUL Prediction

arXiv:2607. 19380v1 Announce Type: new Abstract: Remaining useful life (RUL) prediction estimates how long an engine can continue safe operation and is central to maintenance planning.

By Pu Cheng, Qiang Miao

benchmarks

arXiv AI

Jul 23

Defense Against LLM Backdoors using Critical Neuron Isolation Pruning

arXiv:2607. 19894v1 Announce Type: cross Abstract: Large language models (LLMs) are vulnerable to backdoor attacks, where hidden triggers induce malicious outputs.

By Yuxi Li, Zhibo Zhang, Kailong Wang, Xingshuo Han, Ling Shi, Haoyu Wang

llms fine-tuning efficiency multimodal benchmarks

arXiv Machine Learning

Jul 23

Diffusion ReRoll: Revisable Denoising for Robotic Sequential Prediction

arXiv:2607. 19919v1 Announce Type: cross Abstract: We propose Diffusion ReRoll, a diffusion-based framework for robotic sequential prediction that enables revisable denoising over horizons.

By Seonsoo Kim, Seongil Hong, Jun-Gill Kang

diffusion robotics benchmarks

arXiv AI

Jul 23

When Does Knowledge Distillation Hurt? Reliability-Aware Distillation for Low-Resource Language Summarization

arXiv:2607. 19956v1 Announce Type: cross Abstract: Knowledge distillation (KD) is a standard approach for compressing sequence-to-sequence models, but its per-sample effects are rarely examined.

By Dipto Sumit, Ankan Kumar Roy Srizon, Sadia Khair Rodela, Atia Haque Asha, Mourchona Afrin, Niloy Farhan, Farig Sadeque

nlp fine-tuning efficiency benchmarks safety

arXiv AI

Jul 23

AgentCgroup: Understanding and Controlling OS Resources of AI Agents

arXiv:2602. 09345v3 Announce Type: replace-cross Abstract: AI agents are increasingly deployed in multi-tenant cloud environments, where they execute diverse tool calls within sandboxed containers, each call with distinct resource demands and rapid fluctuations.

By Yusheng Zheng, Jiakun Fan, Quanzhi Fu, Yiwei Yang, Wei Zhang, Andi Quinn

llms agents benchmarks

arXiv AI

Jul 23

Profile-Graph Memory for LLM Agents: Implicit Cross-Entity Traversal through Narrative Profiles

arXiv:2607. 19359v1 Announce Type: new Abstract: Long-term memory is essential for LLM agents that interact across sessions, yet current memory benchmarks primarily evaluate single-hop recall, leaving multi-hop association largely unmeasured.

By Shengtong Zhu

llms rag agents benchmarks

arXiv AI

Jul 23

Opto-ViT-v2: Noise-Resilient On-Chip Fine-Tuning for Photonic Near-Sensor Vision Transformer Accelerators

arXiv:2607. 19421v1 Announce Type: cross Abstract: Silicon-photonic (SiPh) accelerators have emerged as a promising platform for Vision Transformer (ViT) inference by performing matrix multiplications on microring-resonator (MRR) banks with high throughput and energy efficiency.

By Xuming Chen, Deniz Najafi, Mehrdad Morsali, Chengwei Zhou, Zahra Ghanaatianjobzari, Mahdi Nikdast, Shaahin Angizi, Gourav Datta

llms computer-vision fine-tuning efficiency benchmarks

arXiv AI

Jul 23

Physics-Aware Complex-Valued State Space Model with Scattering-Prior Feature Modulation for PolSAR Image Classification

arXiv:2607. 19787v1 Announce Type: cross Abstract: Polarimetric synthetic aperture radar (PolSAR) image classification is a representative task for physics-aware GeoAI, where land-cover semantics are closely coupled with electromagnetic scattering mechanisms.

By Fangyan Zhang, Fan Zhang, Shiqi Zhou, Jun Ni, Carlos L\'opez-Mart\'inez, Qiang Yin

rag computer-vision benchmarks

arXiv Machine Learning

Jul 23

Differentially Private Neural Network Training Under the Hidden State Assumption

arXiv:2407. 08233v3 Announce Type: replace Abstract: Current differentially private learning paradigms face a severe utility bottleneck: DP-SGD degrades performance through noise accumulation over training steps, while aggregation-based approaches such as PATE suffer from data inefficiency due to disjoint data partitioning.

By Ding Chen, Haochen Luo, Xiaofei Wang, Chen Liu

benchmarks safety

arXiv AI

Jul 23

FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads

arXiv:2607. 19349v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as always-on online services, making efficient LLM serving a critical systems challenge.

By Tiancheng Zhang, Shaoyuan Huang, Mingyuan Wang, Yunfeng Zhao, Xiaofei Wang, Wenyu Wang

llms efficiency benchmarks

arXiv AI

Jul 23

RPPNet: Perceptually-Grouped Rhythm-Pitch Primitives for Long-Term Structure Melody Generation via Boundary-Aware Modeling

arXiv:2607. 19776v1 Announce Type: cross Abstract: Existing symbolic music generation models typically use bars as the basic structural unit.

By Tieyao Zhang, Yuke Liu, Jiaxing Yu, Xinda Wu, Kejun Zhang, Genfang Chen

benchmarks

arXiv Machine Learning

Jul 23

Membership Inference Attacks for Unseen Classes

arXiv:2506. 06488v3 Announce Type: replace Abstract: A key tool in developing safe AI models is \emph{data auditing}, i.

By Pratiksha Thaker, Neil Kale, Zhiwei Steven Wu, Virginia Smith

benchmarks safety

arXiv AI

Jul 23

A Systematic Benchmark of Intensity Normalisation Methods for 3D Knee MRI Segmentation and Cross-Domain Generalisability

arXiv:2607. 20028v1 Announce Type: cross Abstract: Robust out-of-the-box performance is essential for the clinical deployment of deep learning models in medical imaging.

By Oliver Mills, Philip Conaghan, Samuel Relton

computer-vision benchmarks

arXiv AI

Jul 23

Hybrid LLM-Guided Search for Quantum Reservoir Architecture Design

arXiv:2607. 19506v1 Announce Type: cross Abstract: Quantum reservoir computing (QRC) uses fixed quantum dynamics as a high-dimensional temporal feature map and trains only a lightweight classical readout.

By Krishna Bhatia, Gautami Sanjay Naik

llms agents benchmarks

arXiv AI

Jul 23

PerfAgent: Profiler-Guided Iterative Refinement for Repository-Level Code Optimization

arXiv:2607. 19653v1 Announce Type: cross Abstract: Large language model (LLM) agents now perform well on correctness-oriented repository-level tasks, including SWE-Bench issue resolution and feature implementation in real codebases.

By Ryan Deng, Yuanzhe Liu, Bastian Lipka, Yao Ma, Xuhao Chen, Tim Kaler, Jatin Ganhotra

llms agents benchmarks

Get the daily briefing