Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,322 stories · RSS feed

arXiv AI

Sep 15

Building Legal Reward Models for Grounding and Abstention

arXiv:2609.14739v1 Announce Type: cross Abstract: Large language models are increasingly used in high-stakes domains such as law, where systems must ground their reasoning in retrieved evidence and a...

By Rilton Franzone, Valentin No\"el, Puyu Wang, Philip Torr, Fabio J. Fehr

llms rag reinforcement-learning benchmarks

arXiv AI

Sep 15

Calibrating Interpretability Instruments Before Trusting Their Verdicts

arXiv:2609.14754v1 Announce Type: cross Abstract: Causal claims about large language model (LLM) internals rest on measurements. Those might include a projection, a cosine, an ablation delta, or an i...

By Orion Reblitz-Richardson

llms benchmarks safety

arXiv AI

Sep 15

TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps

arXiv:2609.14762v1 Announce Type: cross Abstract: Cloud-hosted large language models (LLMs) are increasingly used for root cause analysis (RCA) in AIOps pipelines, but they introduce data privacy ris...

By Rohit Patel, Susil Kumar Mohanty, Jeenal Chaudhary

llms rag efficiency benchmarks safety

arXiv AI

Sep 15

ActGuard: Pre-execution Action Auditing against Indirect Prompt Injection in LLM Agents

arXiv:2609.14987v1 Announce Type: cross Abstract: Large language model (LLM) agents interact with external environments through tool invocation, but tool outputs can also expose them to indirect prom...

By Bingzheng Wang, Xiaoyan Gu, Wentao Wang, Xingyou Yang, Hongcheng Li, Rong Yin

llms agents benchmarks

arXiv AI

Sep 15

PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift

arXiv:2609.15017v1 Announce Type: cross Abstract: Prompt-injection detectors are typically evaluated using aggregate F1 on in-distribution test data, which offers limited insight into behavior under...

By Yusuf Khalid Shire, Sang-Chul Kim

fine-tuning benchmarks safety

arXiv AI

Sep 15

Salesforce Koa: An Enterprise Language Model for Agentic Tool Use

arXiv:2609.15066v1 Announce Type: cross Abstract: We present Salesforce Koa, an enterprise language model built by post-training the open-weight Nemotron-3-Super-120B foundation model with reinforcem...

By Zixiang Chen, Sufeng Niu, Yingchi Liu, Wenting Zhao, Akshara Prabhakar, Shubham Mehrotra, Bin Bi, Zhujun Lan, Katherine Tan, Mohammad Ramezanali, Tulika Manoj Awalgaonkar, Monojit Banerjee, Jielin Qiu, Shiva Kumar Pentyala, Zhepeng Cen, Anupam Tripathi, Ali Ziaei, Regunathan Radhakrishnan, Darvish Lee Shadravan, Shelby Heinecke, Sitaram Asur, Silvio Savarese, James Zhu, Phil Mui, Huan Wang

llms agents reinforcement-learning benchmarks

arXiv AI

Sep 15

Generate to Explore, Select to Exploit: Aligning LLM-based Headline Generation with Personalized Recommendation

arXiv:2609.15094v1 Announce Type: cross Abstract: In industrial recommendation feeds, presenting a static headline for an item often fails to satisfy the diverse, multimodal interests of the user pop...

By Yi Chen, Rufeng Cheng, Qiang Xie, Tao Li

llms multimodal benchmarks

arXiv AI

Sep 15

Failure-Guided Co-Evolution of Prompts and Training Data

arXiv:2609.15209v1 Announce Type: cross Abstract: Automatic prompt optimization (APO) improves language-model programs by revising prompts from task feedback, yet it typically holds its training data...

By Tianyu Yuan, Zhuzhong Qian

benchmarks

arXiv AI

Sep 15

Pre-PEFT Probing: Weight Statistics and Perturbation Robustness for Layer Selection in VLM Vision Encoders

arXiv:2609.15229v1 Announce Type: cross Abstract: We propose a pre-fine-tuning probing method for Parameter-Efficient Fine-Tuning (PEFT) layer selection, aiming to obtain more stable and higher gains...

By Qingtao Xia, Jiahua Bao, Siyao Cheng, Jie Liu

llms fine-tuning multimodal benchmarks

arXiv AI

Sep 15

Clean Scores, Buried Evidence, and Confident Wrong: A Receipt-Based Audit of Frontier Agentic QA

arXiv:2609.15319v1 Announce Type: cross Abstract: Frontier models score well on shallow document/chart reading tasks. In a controlled data-room audit, moving evidence into buried conditions reduced a...

By Luis M. S\'anchez

agents benchmarks safety

arXiv AI

Sep 15

IWC-Bench: Evaluating Web Application Generation from a Software Testing Perspective

arXiv:2609.15387v1 Announce Type: cross Abstract: Human evaluation provides a direct measure of the quality of LLM-generated web applications. However, fitting human judgments through automated evalu...

By Chenxu Liu, Zilu Zou, Peizhong Gao, Jiawen Tao, Zhexin Zhang, Guang Chen, Haowei Lin, Ying Zhou, Tianyi Bai, Dolly Deng, Suncong Zheng, Maxm Pan

llms agents benchmarks safety

arXiv AI

Sep 15

Conflict-Predictive Variable Horizons in Multi-Drone Distributed Model Predictive Control

The paper introduces a conflict‑predictive variable horizon for distributed model predictive control in multi‑drone collision avoidance. Each drone locally estimates future conflicts using observed positions and confidence funnels, then selects the minimal horizon that covers the farthest predicted conflict, shrinking in clear air and expanding only when necessary. The authors prove that this adaptive horizon preserves recursive feasibility and asymptotic stability for linear models, and demonstrate in simulation that it reduces per‑step solver cost and total computation while maintaining separation on dense benchmarks.

By Linda M\"{u}mken, Michael Schwung, Stefan Lier, Andreas Schwung

robotics benchmarks

arXiv AI

Sep 15

Robust and Efficient Communication for Multi-Agent Learning

arXiv:2609.15361v1 Announce Type: cross Abstract: Effective communication is a cornerstone of distributed intelligence in Multi-Agent Reinforcement Learning (MARL), yet ensuring that generated messag...

By Rafael Pina, Varuna De Silva, Corentin Artaud

agents reinforcement-learning robotics benchmarks

arXiv AI

Sep 15

Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation

arXiv:2609.15467v1 Announce Type: cross Abstract: Adding answer options can lower multiple-choice scores without improving assessment validity. Turkish MMLU Pro examines this distinction using 12,000...

By M. Ali Bayram

llms rag agents benchmarks

arXiv AI

Sep 15

Multi-View Molecular Representation Learning with Hierarchical Graphs and Contextualized Fingerprints

arXiv:2609.15611v1 Announce Type: cross Abstract: Molecular property prediction requires representations that generalize from limited labeled data to structurally novel compounds. Existing molecular...

By Gwang-Hyeon Yun, Jong-Hoon Park, Bing Hu, Helen Chen, Anita Layton, Young-Rae Cho

rag fine-tuning benchmarks

arXiv AI

Sep 15

Benchmarking Intra-Patient 3D Deformable Multimodal Image Registration

arXiv:2609.15669v1 Announce Type: cross Abstract: Multimodal image registration is a key component of many clinical workflows, yet it remains challenging because corresponding anatomical structures o...

By Matteo Barbieri, Giammarco La Barbera, Juan Pablo De La Plata, Sabine Sarnacki, Isabelle Bloch, Pietro Gori

multimodal benchmarks safety

arXiv AI

Sep 15

CiteGuard-RAG: A Validation-Centered AI System for Evidence-Grounded Question Answering

arXiv:2609.15830v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) can improve access to complex information; however, retrieving evidence alone does not ensure that answers are g...

By Sumit Barua, Guan Hong, Halil Dursunoglu, Charles Rodgers, Alvis Fong

rag nlp benchmarks safety

arXiv AI

Sep 15

Before You Poll with LLMs: A Deliberative Diagnostic Framework

arXiv:2609.15849v1 Announce Type: cross Abstract: Can LLMs reason through new information like humans, or do they merely retrieve cached opinions? This is critical for silicon sampling, where LLM per...

By Ahmed Wali, Hassaan Tayyab

llms benchmarks

arXiv AI

Sep 15

Privacy-enhanced federated learning via asynchronous aggregation and local differential perturbation

arXiv:2609.15885v1 Announce Type: cross Abstract: This study proposes a privacy-enhanced federated learning framework to address secure collaborative training in distributed data environments. The fr...

By Zhen Zhong (Georgetown University, Washington, D.C., USA), Shini Yang (LinkedIn, CA, USA), Liesheng Wei (Shanghai Ocean University, Shanghai, China)

benchmarks safety

arXiv AI

Sep 15

The Router Within: Eliciting Native Skill Routing from a Frozen LLM

arXiv:2609.15982v1 Announce Type: cross Abstract: Skills extend an LLM agent beyond its parametric knowledge, and the gain they promise rests on picking the right one. Deployed harnesses route by pre...

By Ruishuo Chen, Xun Wang, Yu Chen, Zhuoran Li, Longbo Huang

llms agents benchmarks

Get the daily briefing