Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,257 stories · RSS feed

arXiv AI

Jul 7

Seeing Once is Enough? Online Geometry-Aware Token Pruning for 3D Question Answering

arXiv:2607. 04079v1 Announce Type: cross Abstract: Recent Multi-modal Large Language Models (MLLMs) have demonstrated remarkable performance on 2D question answering tasks.

By Ruei-Chi Lai, Bolivar Solarte, Chin-Hsuan Wu, Yi-Hsuan Tsai, Min Sun

llms nlp efficiency multimodal benchmarks

arXiv AI

Jul 7

Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQL

arXiv:2607. 03833v1 Announce Type: cross Abstract: While Large Language Models (LLMs) have achieved remarkable success in Text-to-SQL tasks, their deployment in real-world environments is hindered by latent reliability issues.

By Hanqing Wang, Yongdong Chi, Jian Yang, Lei Yang, Jiehui Zhao, Yun Chen, Guanhua Chen

llms fine-tuning benchmarks data-engineering

arXiv AI

Jul 7

RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilities

arXiv:2607. 04729v1 Announce Type: cross Abstract: LLM agents are increasingly applied to vulnerability analysis, but existing benchmarks have not kept pace.

By Tarek Elsayed, Shiping Yang, Eunsong Koh, Sanika Goyal, Vincent Huang, Paul Ngo, Nathan Young, Mohammad Omidvar Tehrani, Alvyn Kang, Arnell Kang, Zeyu Chen, Ang\'elica Moreira, Xuan Feng, Angel X. Chang, Nick Sumner, Steven Y. Ko

llms agents benchmarks safety

arXiv AI

Jul 7

Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions

arXiv:2607. 04072v1 Announce Type: cross Abstract: Large language models can generate plausible quantum code, but it is unclear whether they can reliably target the specific software development kit (SDK) version requested by the user.

By Mohammad Arif Rasyidi, Syahirul Faiz

llms benchmarks safety

arXiv AI

Jul 7

DrugAgent: Reliable Multi-Agent Integration of Conflicting Biomedical Evidence for Drug-Target Interaction Assessment

arXiv:2408. 13378v5 Announce Type: replace Abstract: Workflows in drug-target interaction (DTI) assessment require integrating heterogeneous data from predictive models, curated resources, and observations from experimental literature.

By Yoshitaka Inoue, Tianci Song, Xinling Wang, Rui Kuang, Tianfan Fu, Augustin Luna

llms rag agents nlp benchmarks

arXiv AI

Jul 7

Out-of-Distribution Generalization of Risk Aversion in Language Models

arXiv:2607. 02755v1 Announce Type: cross Abstract: Training AIs to be risk-averse in resources could offer a failsafe in the event that AIs turn out misaligned.

By Kristina Zhang, Junior Chinomso Okoroafor, Benjamin Maltbie, Andrew Lin, Abhitej Bokka, Elliott Thornley

llms reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 7

Evaluating LLM-Based Regression Test Generation

arXiv:2501. 11086v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have shown tremendous promise in automated software engineering.

By Jing Liu, Seongmin Lee, Eleonora Losiouk, Marcel B\"ohme

llms nlp benchmarks

arXiv AI

Jul 7

NormWorlds-CF: Solver-Verified Counterfactual Normative Reasoning with Metamorphic-Relation GRPO

arXiv:2607. 03957v1 Announce Type: cross Abstract: Language models can reach the right normative verdict for the wrong reason.

By Xinqi Zhang

llms fine-tuning benchmarks

arXiv AI

Jul 7

Governed Individuation: Cryptographically Decoupling an Agent's Learning from Its Authority

arXiv:2607. 04613v1 Announce Type: new Abstract: Autonomous agents are moving from sandboxed text generators to operators of code, data, and physical infrastructure, and they increasingly learn while deployed.

By Xue Qin, Simin Luan, Cong Yang, Zhijun Li

agents benchmarks safety

arXiv AI

Jul 7

Correlation-Weighted Multi-Reward Optimization for Compositional Generation

arXiv:2603. 18528v2 Announce Type: replace Abstract: Text-to-image models produce images that align well with natural language prompts, but compositional generation has long been a central challenge.

By Jungmyung Wi, Hyunsoo Kim, Donghyun Kim

diffusion benchmarks

arXiv AI

Jul 7

Measuring Harness-Induced Belief Divergence in Multi-Step LLM Agents

arXiv:2607. 04528v1 Announce Type: new Abstract: Software-agent benchmarks usually report whether an agent solves a task, but the agent reaches that outcome through a harness that controls what it sees, which actions it can take, which failures are repaired, which states are verified, and which evidence is logged.

By Haiwen Yi, Xinyuan Song

llms agents efficiency benchmarks safety

arXiv AI

Jul 7

Criterion-Conditional In-Context Learning: Evaluating Criterion-Shift Adaptation in Vision-Language Models

arXiv:2607. 02575v1 Announce Type: cross Abstract: Vision-language models can perform new tasks without parameter updates through in-context learning (ICL), whose core mechanism is utilizing the support set for task induction.

By Kaiyun Yang, Ruilin Yang, Zhimin Yao, J. Wang, Wei Ge

llms multimodal benchmarks safety

arXiv Machine Learning

Jul 7

CodeJeNN: A simple C++ neural network generator for physics applications

arXiv:2607. 02746v1 Announce Type: cross Abstract: Machine learning has shown speedups for numerical methods in physics applications, but integrating Python-based libraries into high-performance C++ solvers creates performance bottlenecks.

By Jay Arcities, Pavel Popov, Eric J Ching, Kamal Viswanath, Ryan F Johnson

benchmarks

arXiv AI

Jul 7

Bridging Interleaved Multi-Modal Reasoning as a Unified Decision Process

arXiv:2607. 03748v1 Announce Type: new Abstract: Unified multi-modal models (UMMs) have shown promising interleaved text-image reasoning capabilities, yet effectively optimizing such multi-turn generation via reinforcement learning (RL) remains an open challenge.

By Zican Hu, Xuyang Hu, Yiming Liu, Zuwei Long, Wei Liu, Yunzhuo Hao, Jiawei Gu, Linjie Li, Yu Cheng, Zhenhong Sun, Weibo Gu, Xing Sun, Zhi Wang

llms diffusion reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jul 7

Poisson-Gamma Modeling of Inter-Relational Dependencies in Dynamic Knowledge Graphs

arXiv:2607. 02872v1 Announce Type: new Abstract: Dynamic knowledge graphs are ubiquitous in today's AI applications, as we represent molecular structures, social relationships, and language information using these graph models.

By Nan Fang, Yijun Wang, Hao Liao, Sikun Yang

benchmarks

arXiv Machine Learning

Jul 7

Air-Plan: Query-Optimized Topology Selection for Over-the-Air Decentralized Federated Learning

arXiv:2607. 04254v1 Announce Type: cross Abstract: Over-the-air (OTA) aggregation exploits the superposition property of wireless multiple-access channels to aggregate model updates from multiple devices within a single transmission slot, significantly reducing communication latency.

By Kaushal Attaluri, Rebeca P. Diaz-Redondo, Manuel Fernandez Veiga

benchmarks safety

arXiv AI

Jul 7

Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

arXiv:2607. 03784v1 Announce Type: cross Abstract: While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size.

By Zhenfeng Su, Kang Zhao, Han Bao, Tao Yuan, Zhongzhe Hu, Xianzhi Yu, Wenxuan Wang

llms efficiency benchmarks

arXiv AI

Jul 7

PDFBench: A Benchmark for De novo Protein Design from Function

arXiv:2505. 20346v3 Announce Type: replace-cross Abstract: Function-guided protein design is a crucial task with significant applications in drug discovery and enzyme engineering.

By Jiahao Kuang, Nuowei Liu, Jie Wang, Changzhi Sun, Tao Ji, Yuanbin Wu

benchmarks

arXiv AI

Jul 7

HVR-Met: A Hypothesis-Verification-Replanning Agentic System for Extreme Weather Diagnosis

arXiv:2603. 01121v2 Announce Type: replace Abstract: While deep learning-based weather forecasting paradigms have made significant strides, addressing extreme weather diagnostics remains a formidable challenge.

By Shuo Tang, Jiadong Zhang, Gengxian Zhou, Qizhao Jin, Qinxuan Wang, Yi Hu, Ning Hu, Hongchang Ren, Lingli He, Shiming Xiang, Jingtao Ding, Jian Xu, Jiaolan Fu, Cheng-Lin Liu

agents benchmarks

arXiv AI

Jul 7

AutoCedar: An Agentic Framework for Verifier-Guided Access Control Policy Synthesis

arXiv:2607. 03656v1 Announce Type: cross Abstract: Large Language Models are increasingly used to turn natural-language requirements into code.

By Adarsh Vatsa, Sachi Shome, Yingming Zhou, William Eiers

llms agents benchmarks

Get the daily briefing