Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,829 stories · RSS feed

arXiv AI

Jul 23

Sound Probabilistic Safety Bounds for Large Language Models

arXiv:2607. 20286v1 Announce Type: cross Abstract: We propose a novel framework for computing rigorous bounds on the probability that a large language model (LLM) generates harmful output to a given prompt.

By Mahdi Nazeri, Anne-Kathrin Schmuck, Sadegh Soudjani, Alessandro Abate

llms diffusion benchmarks safety

arXiv AI

Jul 23

Closing the Lab-to-Store Gap: A Data-Efficient Post-Training and Experience-Driven Learning VLA Framework for Retail Humanoids

arXiv:2607. 20345v1 Announce Type: cross Abstract: Closing the gap between benchmark performance and reliable real-world operation remains a central challenge for Vision-Language-Action (VLA) humanoid robots, which must handle execution errors, distribution shifts, and environmental variability.

By Roger Sala Sis\'o, Tiago Silv\'erio, Jakob Sand, Tran Nguyen Le

robotics fine-tuning multimodal benchmarks safety

arXiv AI

Jul 23

G-MAD: A Game-Based Data Generation Framework for Multi-View RGB-T Aerial Object Detection

arXiv:2607. 19942v1 Announce Type: cross Abstract: This work introduces G-MAD, an open-source framework that uses Arma3 to generate synchronized multi-view RGB-T data for aerial object detection.

By Yechan Kim, JongHyun Park, Dongho Yoon, Namhoon Jung, Moongu Jeon

computer-vision multimodal benchmarks safety

arXiv AI

Jul 23

AgentCgroup: Understanding and Controlling OS Resources of AI Agents

arXiv:2602. 09345v3 Announce Type: replace-cross Abstract: AI agents are increasingly deployed in multi-tenant cloud environments, where they execute diverse tool calls within sandboxed containers, each call with distinct resource demands and rapid fluctuations.

By Yusheng Zheng, Jiakun Fan, Quanzhi Fu, Yiwei Yang, Wei Zhang, Andi Quinn

llms agents benchmarks

arXiv AI

Jul 23

FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation

arXiv:2607. 19354v1 Announce Type: new Abstract: Spreadsheet applications are used by hundreds of millions worldwide, yet writing formulas remains a significant barrier.

By Cy Xie

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 23

Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results

arXiv:2607. 20090v1 Announce Type: cross Abstract: Retrieval-augmented large language models frequently face contexts that interleave useful evidence with misleading statements or instruction-like content.

By Yanyu Chen, Yue Li, Yongyi Cui, Dongsheng Shi, Lichang Dai

llms rag reinforcement-learning benchmarks

arXiv AI

Jul 23

Pushing the Frontier of Full-Song Generation: Hierarchical Autoregressive Planning Meets Flow-Matching Rendering

arXiv:2607. 20253v1 Announce Type: cross Abstract: In this report, we present a unified song generation framework capable of producing high-quality full-length music from lyrics, text descriptions, and musical attributes.

By Junyu Dai, Xinyue Fan, Weiqin Li, Xiangang Li, Yunjia Li, Bin Ma, Yukun Ma, Chongjia Ni, Yufei Shi, Haoxu Wang, Menglin Wu, Jianwei Yu, Huaicheng Zhang, Han Zhao, Shengkui Zhao, Haina Zhu

diffusion nlp reinforcement-learning benchmarks

arXiv AI

Jul 23

LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

arXiv:2607. 19391v1 Announce Type: cross Abstract: Low-Rank Adaptation is widely used for parameter-efficient fine-tuning, yet existing methods typically assign the same adapter rank to every transformer layer despite their heterogeneous adaptation requirements.

By Ashutosh Tripathi, Surya Deep Singh, Pranab Sahoo, Sriparna Saha

llms fine-tuning benchmarks

arXiv Machine Learning

Jul 23

CruiseBench: A Real-Flight-Aligned N-CMAPSS Benchmark for Engine RUL Prediction

arXiv:2607. 19380v1 Announce Type: new Abstract: Remaining useful life (RUL) prediction estimates how long an engine can continue safe operation and is central to maintenance planning.

By Pu Cheng, Qiang Miao

benchmarks

arXiv AI

Jul 23

PRIME-SVR: Physics-infoRmed Implicit Multi-Echo Slice-to-Volume Reconstruction for Fetal T2 mapping

arXiv:2607. 20136v1 Announce Type: cross Abstract: Slice-to-volume reconstruction (SVR) is the standard method for obtaining high-resolution (HR) 3D fetal brain volumes from motion-corrupted 2D MRI slice stacks acquired in multiple orientations.

By Busra Bulut, Maik Dannecker, Thomas Sanchez, Sara Neves Silva, Steven Jia, Jean-Baptiste Ledoux, Leo Pomar, Joanna Sichitiu, Yvan Gomez, Meriam Koob, Vincent Dunet, Maria Deprez, Guillaume Auzias, Francois Rousseau, Jana Hutter, Daniel Rueckert, Meritxell Bach Cuadra

benchmarks

arXiv AI

Jul 23

The Chronos Vulnerability: A Taxonomy of Temporal Persistence and Memory-Based Deception in Agentic AI

arXiv:2607. 19433v1 Announce Type: new Abstract: The transition from stateless generative models in artificial intelligence to stateful, autonomous agents represents an architectural evolution that, while providing the capabilities of long-term planning and the automation of enterprise workflows, also represents the introduction of a new form of security threat, the Chronos Vulnerability.

By Om Narayan, Ramkinker Singh, Praveen Baskar

agents benchmarks safety

arXiv AI

Jul 23

Language-Specific versus Cross-Lingual Knowledge Graphs for Implicit Aspect Identification in Arabic: A Comparative Study of Reasoning and Adaptation Strategies

arXiv:2607. 20056v1 Announce Type: cross Abstract: Aspect-based sentiment analysis (ABSA) in Arabic must recover both explicitly stated aspects and implicit aspects that are never named in the text.

By Lujain A. Alawwad

llms rag nlp fine-tuning benchmarks

arXiv Machine Learning

Jul 23

Material-agnostic temperature field prediction for metal additive manufacturing via a parametric PINN framework

arXiv:2604. 14562v2 Announce Type: replace Abstract: Accurate temperature field prediction in metal additive manufacturing (AM) is essential for understanding the process-structure-performance relationship.

By Hyeonsu Lee, Jihoon Jeong

benchmarks

arXiv AI

Jul 23

Generative Augmented Inference of LLM-generated Data for Market Research: Theory and Empirical Evidence

arXiv:2604. 14575v3 Announce Type: replace-cross Abstract: Marketing research often relies on parameters estimated from costly human-generated data, such as conjoint survey responses, purchase decisions, and field experiment outcomes.

By Cheng Lu, Mengxin Wang, Dennis J. Zhang, Heng Zhang

llms benchmarks

arXiv AI

Jul 23

ENTRAP-VL: A Taxonomic Probe for Dual Contextual Entrainment in Vision-Language Models

arXiv:2607. 20092v1 Announce Type: cross Abstract: Contextual entrainment is the tendency of a model to let auxiliary context in its input pull its output, independently of whether that context is relevant, true, or even meaningful.

By Karan Goyal, Afreen Hossain, Debojyoti Das, Vishal Bhutani

llms multimodal benchmarks

arXiv Machine Learning

Jul 23

TempoNet: Slack-Quantized Transformer-Guided Reinforcement Scheduler for Adaptive Deadline-Centric Real-Time Dispatchs

arXiv:2602. 18109v3 Announce Type: replace Abstract: Real-time schedulers must reason about tight deadlines under strict compute budgets.

By Rong Fu, Yibo Meng, Zeyu Zhang, Ziming Guo, Jia Yee Tan, Xiaojing Du, Simon James Fong

llms rag nlp reinforcement-learning efficiency benchmarks

arXiv AI

Jul 23

SCPP: A Unified Python Library for Soft Clustering

arXiv:2607. 19620v1 Announce Type: cross Abstract: In this paper, we present SCPP (Soft Clustering Python Package), an open-source Python framework for soft clustering.

By Kiyan Rezaee, Morteza Ziabakhsh, Artin Bahrampour, Seyed Mohammad Ghoreishi, Asal Khaje, Ali Sajedifar, Manny Chalak, Ava Zerafatangiz, Sadegh Eskandari

benchmarks

arXiv Machine Learning

Jul 23

Multi-stage Dynamic Selection for Cross-Project Defect Prediction

arXiv:2607. 20151v1 Announce Type: cross Abstract: Cross-Project Defect Prediction (CPDP) involves building models using data from external projects, called training projects, to predict modules from the target project.

By Juscimara G. Avelino, Juscelino S. A. Junior, George D. C. Cavalcanti, Rafael M. O. Cruz

benchmarks

arXiv Machine Learning

Jul 23

Enhanced Neural Quantum State via Annealed Gradient Descent

arXiv:2607. 18865v2 Announce Type: replace-cross Abstract: Neural quantum states offer expressive representations of quantum many-body wave functions, yet their practical accuracy can be limited by stochastic optimization rather than representational capacity.

By Shiwei Zhou, Yiming Huang, Xiao Yuan, Xiaoxia Cai

benchmarks

arXiv AI

Jul 23

EvoDRC: A Self-Evolving Agentic Framework for Automated DRC Violation Repair

arXiv:2607. 20019v1 Announce Type: new Abstract: Design rule check (DRC) closure remains a major bottleneck in advanced-node physical design.

By Bing-Yue Wu, Chia-Tung Ho, Haoyu Yang, Brucek Khailany, Vidya A. Chhabria

llms agents benchmarks

Get the daily briefing