Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

17,521 stories · RSS feed

arXiv AI

Jul 8

LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding

arXiv:2607. 05769v1 Announce Type: cross Abstract: We propose a novel pipeline, Legato 2, for extracting symbolic notation and semantic knowledge from images of sheet music.

By Guang Yang, Brian Siyuan Zheng, Victoria Ebert, Noah A. Smith

llms computer-vision multimodal benchmarks

arXiv AI

Jul 8

The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities

arXiv:2607. 05743v1 Announce Type: cross Abstract: AI coding agents now read repositories, call tools, and execute shell commands with limited human oversight, and a fast-growing body of work studies whether the execution layer around them is actually safe.

By Mohammadreza Rashidi

llms agents benchmarks safety

arXiv AI

Jul 8

Narrative World Model: Narratology-Grounded Writer Memory for Long-Form Fiction

arXiv:2607. 05577v1 Announce Type: new Abstract: Long-form fiction writers need memory that answers multi-hop questions about evolving story state: who knows a secret and when they learned it, whether an event preceded the narration that revealed it, whether a setup paid off, and how a relationship shifted.

By Mohammad Saifullah, Thomas Kornmaier, Taaha Kazi, Vasu Sharma, Aditya Sanjiv Kanade, Aanand Kumar Yadav

agents benchmarks

arXiv AI

Jul 8

Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents

arXiv:2607. 05775v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and operate over extended horizons.

By Wael Albayaydh, Rui Zhao, Ivan Flechais

llms agents benchmarks safety

arXiv AI

Jul 8

Learning to Control LLM Agent Harnesses with Offline Reinforcement Learning

arXiv:2607. 05458v1 Announce Type: cross Abstract: Large language model (LLM) agents are usually improved by changing prompts, models, or hand-written workflows, while the execution harness around the model is treated as fixed infrastructure.

By Haiwen Yi, Xinyuan Song

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 8

PolicyShiftGuard: Benchmarking and Improving Policy-Adaptive Image Guardrails

arXiv:2607. 05910v1 Announce Type: cross Abstract: Image guardrails are typically trained and evaluated under a fixed safety policy, implicitly treating safety as an intrinsic property of an image.

By Mingyang Song, Luxin Xu, Haoyu Sun, Minzhou Pan, Yu Cheng, Bo Li

fine-tuning benchmarks safety

arXiv Machine Learning

Jul 8

Federated Physics-Grounded Reinforcement Learning for Distributed Stability Control in Smart Grids

arXiv:2607. 05553v1 Announce Type: new Abstract: Transient stability control in smart grids requires rapid post-fault damping of generator frequency and rotor angle deviations to prevent cascading failures.

By Omar Al-Refai, Ibrahim Shahbaz, Adam Ali Husseinat, Eman Hammad

agents reinforcement-learning efficiency benchmarks

arXiv AI

Jul 8

Integrating knowledge graphs and multilingual scholarly corpora for domain-adaptive LLMs in SSH

arXiv:2607. 05956v1 Announce Type: new Abstract: The integration of Large Language Models (LLMs) into scientific research workflows, particularly for bibliographic discovery and literature synthesis, raises significant methodological, epistemic and regulatory challenges for the Social Sciences and Humanities (SSH), especially with regard to disciplinary diversity, multilingual access to sources and the evaluation of results.

By Adam Faci, Alessio Miaschi, Anne Combe, Pascal Cuxac, Francesca Frontini, Nicolas Larrousse, St\'ephane Pouyllau

llms rag nlp benchmarks safety

arXiv Machine Learning

Jul 8

Fine-Tuning Regimes Define Distinct Continual Learning Problems

arXiv:2604. 21927v3 Announce Type: replace Abstract: Continual learning (CL) studies how models acquire tasks sequentially while retaining previously learned knowledge.

By Paul-Tiberiu Iordache, Elena Burceanu

fine-tuning benchmarks

arXiv AI

Jul 8

Token-Based Dual-view Fusion and Adaptation of Large Vision Models for Breast Cancer Classification

arXiv:2607. 06309v1 Announce Type: cross Abstract: Accurate breast cancer classification from mammography requires effective integration of complementary information from craniocaudal (CC) and mediolateral oblique (MLO) views, which provide a more complete characterization of breast abnormalities.

By Aysan Ghayouri Pirsoltan, Shima Babakordi, Mohammad Reza Mohammadi

llms computer-vision benchmarks

arXiv AI

Jul 8

RPAM: A Principled Metric for Evaluating Associations in Language Models with High Predictive Validity in Downstream Outputs

arXiv:2607. 05679v1 Announce Type: cross Abstract: Language models (LMs) exhibit problematic biases, such as stereotypes.

By Damian Hodel, Jevin West, Aylin Caliskan

llms rag benchmarks

arXiv Machine Learning

Jul 8

EntroPath: Maximum Entropy Path Ensemble Embedding for Manifold Learning

arXiv:2607. 06497v1 Announce Type: new Abstract: We introduce EntroPath, a manifold learning method that recovers geodesic geometry from data graphs through ensembles of diffusion paths.

By Przemys{\l}aw Rola

rag diffusion benchmarks

arXiv AI

Jul 8

AbICL: In-Context Learning for Antigen-Specific Antibody Affinity Ranking

arXiv:2607. 05846v1 Announce Type: cross Abstract: Accurate ranking of antibody candidates according to their binding affinity is essential for therapeutic antibody discovery.

By Zhiyuan Chen, Jing Hu, Junzhe Wang, Yueyang Huang, Xinyi Yang, Zhaoyang Wang, Feng Zhu

llms benchmarks

arXiv AI

Jul 8

An Experimental Design Approach to Evaluating Agentic AI's Autonomous Model Discovery

arXiv:2607. 06413v1 Announce Type: cross Abstract: Large language model coding agents increasingly perform open-ended data modeling and analysis.

By Hao He, Xueying Liu, Chris J. Kuhlman, Xinwei Deng

llms agents benchmarks

arXiv AI

Jul 8

Evaluating calibrated refusal and safe usefulness in dual-use biology settings

arXiv:2607. 05462v1 Announce Type: cross Abstract: As AI agents are incorporated into life science workflows, the capabilities that speed discovery might also enable misuse.

By Edwin H. Wintermute, Harmon Bhasin, Christina M. Agapakis, Dianzhuo Wang, Evan Seeyave, Arjun Banerjee, Daniel Fulop, Matthew C. Watson, Adam J. Meyer, Sandrine Boissel, Jens H. Kuhn, Rishi Jain, Noah D. Taylor, Helena Shomar, Patrick M. Boyle, Kenny Workman

agents benchmarks

arXiv AI

Jul 8

KAT-Coder-V2.5 Technical Report

arXiv:2607. 05471v1 Announce Type: cross Abstract: We present KAT-Coder-V2.

By Bo Huang, Fengxiang Li, Hao Xu, Haoyang Huang, Hongyi Fu, Jinhua Hao, Kun Yuan, Minglei Zhang, Pengcheng Xu, Shiyang Liu, Wenhao Zhuang, Yuze Shi, Zongxian Feng, Chao Wang, Cheng He, Chongling Rao, Deyu Cao, Fan Yang, Gang Xiong, Haochen Liu, Jiabao Li, Jian Liang, Jinghui Jia, Jingwen Chang, Jun Du, Junyu Shi, Min Li, Mingqi Wu, Qiang Gao, Shangpeng Yan, Shaotong Qi, Shu Xu, Shuo Zhou, Tiankuo Xu, Tong Zheng, Weilun Zhao, Xiancheng Meng, Xianda Sun, Xiaoyu Jiang, Xunhao Jia, Yao Xia, Yimeng Xu, Yinghan Cui, Yingpeng Chen, Yiwen Ning, Yong Wang, Yuxuan Sun, Zhongsheng Liu, Ming Sun, Cheng Luo, Chen Yang, Han Li, Kun Gai

agents reinforcement-learning efficiency benchmarks

arXiv AI

Jul 8

Information Gain-based Rollout Policy Optimization: An Adaptive Tree-Structured Rollout Approach for Multi-Turn LLM Agents

arXiv:2607. 06223v1 Announce Type: new Abstract: Reinforcement learning has become a promising paradigm for improving large language model (LLM) agents on long-horizon search tasks, where the agent must make a sequence of intermediate decisions before receiving a final outcome.

By Yijun Zhang, Fan Xu, Jiaxin Ding, Yule Xie, Shiqing Gao, Xin Ding, Haoxiang Zhang, Luoyi Fu, Xinbing Wang

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 8

Supervised Reward Inference

arXiv:2502. 18447v2 Announce Type: replace Abstract: Existing approaches to reward inference typically assume that humans provide demonstrations according to specific behavior models.

By Will Schwarzer, Jordan Schneider, Philip S. Thomas, Scott Niekum

robotics benchmarks

arXiv AI

Jul 8

Replication in Visual Diffusion Models: A Survey and Outlook

arXiv:2408. 00001v2 Announce Type: replace-cross Abstract: Visual diffusion models have revolutionized the field of creative AI, producing high-quality and diverse content.

By Wenhao Wang, Yifan Sun, Zongxin Yang, Zhengdong Hu, Zhentao Tan, Yi Yang

diffusion benchmarks safety

arXiv Machine Learning

Jul 8

PGOT: A Physics-Geometry Operator Transformer for Complex PDEs

arXiv:2512. 23192v4 Announce Type: replace Abstract: While Transformers have demonstrated remarkable potential in modeling Partial Differential Equations (PDEs), modeling large-scale unstructured meshes with complex geometries remains a significant challenge.

By Zhuo Zhang, Xi Yang, Ying Miao, Xiaobin Hu, Yifu Gao, Yong Yang, Canqun Yang, Boocheong Khoo

llms benchmarks

Get the daily briefing