Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,065 stories · RSS feed

arXiv Machine Learning

Jul 23

Marine Engine Fault Dataset: Open-Access Data under Controlled Reference and Fault Scenario Conditions

arXiv:2607. 19444v1 Announce Type: new Abstract: Open-access datasets for marine-engine predictive maintenance remain scarce, particularly those from controlled fault experiments with documented operating conditions, subsystem-level interventions and system-level measurements.

By Ahmad BahooToroody, Oleksiy Bondarenko, Mohammad Mahdi Abaei, Niki Yoichi, Enrico Zio

benchmarks

arXiv AI

Jul 23

OPIUM: Mitigating Steering Externalities and Over-Refusal via Dual Objective Latent Optimization

arXiv:2607. 19806v1 Announce Type: cross Abstract: Activation steering provides a lightweight mechanism for controlling large language models at inference time, but steering vectors can have unintended externalities: utility vectors may weaken safety behavior, while refusal vectors may induce over-refusal on benign prompts.

By Kavin Aravindan, Arihant Rastogi, Aadi Prasad, Krishak Aneja, Saiyam Jain, Vaishnavi Shivkumar, Ponnurangam Kumaraguru

llms benchmarks safety

arXiv AI

Jul 23

AgentCgroup: Understanding and Controlling OS Resources of AI Agents

arXiv:2602. 09345v3 Announce Type: replace-cross Abstract: AI agents are increasingly deployed in multi-tenant cloud environments, where they execute diverse tool calls within sandboxed containers, each call with distinct resource demands and rapid fluctuations.

By Yusheng Zheng, Jiakun Fan, Quanzhi Fu, Yiwei Yang, Wei Zhang, Andi Quinn

llms agents benchmarks

arXiv Machine Learning

Jul 23

Missing-by-Design: Certifiable Modality Deletion for Revocable Multimodal Sentiment Analysis

arXiv:2602. 16144v4 Announce Type: replace-cross Abstract: As multimodal systems increasingly process sensitive personal data, the ability to selectively revoke specific data modalities has become a critical requirement for privacy compliance and user autonomy.

By Rong Fu, Ziming Wang, Chunlei Meng, Jiekai Wu, Kangan Qian, Hao Zhang, Simon Fong

rag nlp multimodal benchmarks safety

arXiv AI

Jul 23

Fidelity Before Structure: Verbatim Chunks Beat Lossy Artifact Extraction in Long-Conversation LLM Memory

arXiv:2601. 00821v4 Announce Type: replace Abstract: A growing class of conversational-memory systems compresses dialogue history into structured artifacts (extracted facts, decisions, or events) on the premise that distilled structure retrieves better than raw text.

By Tao An

llms rag efficiency benchmarks

arXiv Machine Learning

Jul 23

SUM: Unified Geometric Surgery on Spatio-Temporal Adaptation Vectors for Federated Class Incremental Learning

arXiv:2607. 19384v1 Announce Type: new Abstract: Real-world intelligent systems often require both distributed collaboration across data-isolated clients and continual adaptation to evolving tasks.

By Jaeik Kim, Jaeyoung Do

benchmarks

arXiv AI

Jul 23

HyGRL: Adaptive Hybrid Graph Reasoning for Multi-Entity Questions

arXiv:2607. 19398v1 Announce Type: new Abstract: Multi-entity compositional questions pose significant challenges to existing retrieval-augmented language models.

By Junyi Wang

llms rag reinforcement-learning efficiency benchmarks

arXiv AI

Jul 23

Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

arXiv:2604. 06996v2 Announce Type: replace-cross Abstract: LLM-as-a-judge has become the de facto approach for evaluating LLM outputs.

By Jos\'e Pombal, Ricardo Rei, Andr\'e F. T. Martins

llms benchmarks safety

arXiv AI

Jul 23

Stateful Guardrails for Multi-Turn LLM Systems: A Conversational Risk Accumulation Framework

arXiv:2607. 19361v1 Announce Type: cross Abstract: Most safety guardrails for large language models (LLMs) evaluate each prompt-response pair in isolation, which misses failures that arise only over a dialogue as benign turns compose into harm.

By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik

llms benchmarks safety

arXiv AI

Jul 23

EvoDRC: A Self-Evolving Agentic Framework for Automated DRC Violation Repair

arXiv:2607. 20019v1 Announce Type: new Abstract: Design rule check (DRC) closure remains a major bottleneck in advanced-node physical design.

By Bing-Yue Wu, Chia-Tung Ho, Haoyu Yang, Brucek Khailany, Vidya A. Chhabria

llms agents benchmarks

arXiv AI

Jul 23

FORCE-Bench: A Benchmark, Dataset, and Evaluation Harness for Agentic AI in Enterprise Finance

arXiv:2607. 19409v1 Announce Type: new Abstract: Recent advances in large language models have accelerated deployment of agentic systems in operational finance.

By Wolfgang M. Pauli, Sarah Panda, Kidus Admassu, Said Bleik, Ademola Okerinde, Jeremy Reynolds

llms agents benchmarks safety

arXiv Machine Learning

Jul 23

Enhanced Neural Quantum State via Annealed Gradient Descent

arXiv:2607. 18865v2 Announce Type: replace-cross Abstract: Neural quantum states offer expressive representations of quantum many-body wave functions, yet their practical accuracy can be limited by stochastic optimization rather than representational capacity.

By Shiwei Zhou, Yiming Huang, Xiao Yuan, Xiaoxia Cai

benchmarks

arXiv AI

Jul 23

NeuroSymActive: Differentiable Neural-Symbolic Reasoning with Active Exploration for Knowledge Graph Question Answering

arXiv:2602. 15353v3 Announce Type: replace-cross Abstract: Large pretrained language models and neural reasoning systems have advanced many natural language tasks, yet they remain challenged by knowledge-intensive queries that require precise, structured multi-hop inference.

By Rong Fu, Yang Li, Zeyu Zhang, Jiekai Wu, Yaohua Liu, Shuaishuai Cao, Yangchen Zeng, Yuhang Zhang, Xiaojing Du, Simon Fong

llms rag nlp benchmarks

arXiv Machine Learning

Jul 23

Label-Free Finite-Volume-Residual Training of Attention Graph Neural Networks for Coupled Thermo-Fluid Fields

arXiv:2607. 20321v1 Announce Type: cross Abstract: Neural surrogates are widely used in scientific machine learning for fast prediction of three-dimensional (3D) thermo-fluid fields.

By Tianyu Li, Zhiwei Cao, Qingang Zhang, Ruihang Wang, Binyang Song, Yonggang Wen

benchmarks

arXiv Machine Learning

Jul 23

Efficient Clustering with Provable Guardrails for LLM Inference at Scale

arXiv:2607. 19704v1 Announce Type: new Abstract: Scaling LLM-based applications to millions of users is bottlenecked by the inference cost and latency of modern foundation models.

By Longshaokan Wang, Wai Tsang Keung, Punit Ghodasara, Roman Wang, Ali Dashti, Francesc Moreno-Noguer

llms rag benchmarks safety

arXiv AI

Jul 23

Language-Specific versus Cross-Lingual Knowledge Graphs for Implicit Aspect Identification in Arabic: A Comparative Study of Reasoning and Adaptation Strategies

arXiv:2607. 20056v1 Announce Type: cross Abstract: Aspect-based sentiment analysis (ABSA) in Arabic must recover both explicitly stated aspects and implicit aspects that are never named in the text.

By Lujain A. Alawwad

llms rag nlp fine-tuning benchmarks

arXiv AI

Jul 23

Defense Against LLM Backdoors using Critical Neuron Isolation Pruning

arXiv:2607. 19894v1 Announce Type: cross Abstract: Large language models (LLMs) are vulnerable to backdoor attacks, where hidden triggers induce malicious outputs.

By Yuxi Li, Zhibo Zhang, Kailong Wang, Xingshuo Han, Ling Shi, Haoyu Wang

llms fine-tuning efficiency multimodal benchmarks

arXiv AI

Jul 23

FormulaSPIN: Self-Play Fine-Tuning for Natural Language to Spreadsheet Formula Generation

arXiv:2607. 19354v1 Announce Type: new Abstract: Spreadsheet applications are used by hundreds of millions worldwide, yet writing formulas remains a significant barrier.

By Cy Xie

reinforcement-learning fine-tuning benchmarks

arXiv Machine Learning

Jul 23

Multi-stage Dynamic Selection for Cross-Project Defect Prediction

arXiv:2607. 20151v1 Announce Type: cross Abstract: Cross-Project Defect Prediction (CPDP) involves building models using data from external projects, called training projects, to predict modules from the target project.

By Juscimara G. Avelino, Juscelino S. A. Junior, George D. C. Cavalcanti, Rafael M. O. Cruz

benchmarks

arXiv Machine Learning

Jul 23

Towards Practical Emotion Recognition: An Unsupervised Source-Free Approach for EEG Domain Adaptation

arXiv:2504. 03707v2 Announce Type: replace-cross Abstract: Emotion recognition is crucial for advancing mental health, healthcare, and technologies such as brain-computer interfaces.

By Md Niaz Imtiaz, Naimul Khan

fine-tuning benchmarks safety

Get the daily briefing