Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,794 stories · RSS feed

arXiv AI

Jul 31

OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

arXiv:2607. 27155v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly expected to assist users in completing tasks.

By Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu, Yanru Ma, Yixiong Xiao, Zenghui Zhang, Le Zhang, Hua Wu

llms agents benchmarks safety

arXiv AI

Jul 31

IDP AutoOpt: Agent-Driven Optimization of Document Processing Pipeline Configurations

arXiv:2607. 26075v1 Announce Type: cross Abstract: We present IDP AutoOpt, an autonomous LLM agent that discovers high-performing configurations for intelligent document processing (IDP) pipelines.

By David Kaleko, Sergey Ivanov, Md Mofijul Islam

llms rag agents computer-vision benchmarks

arXiv AI

Jul 31

Multi-Agent Debate Strategies: Survey, Taxonomy, and Challenges

arXiv:2607. 26212v1 Announce Type: cross Abstract: Multi-Agent Debate (MAD) is a promising paradigm for improving the accuracy and robustness of Large Language Model (LLM)-based agentic systems.

By Quim Motger, Marc Oriol, Jordi Marco, Xavier Franch

llms agents benchmarks

arXiv AI

Jul 31

StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents

arXiv:2607. 26314v1 Announce Type: cross Abstract: Stealth, the discipline of achieving an objective without revealing your presence, capabilities, or collected intelligence, is what separates sophisticated operators from detectable ones.

By Ads Dawson, Adrian Wood

llms rag agents benchmarks

arXiv AI

Jul 31

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

arXiv:2607. 26348v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions.

By Zihan Chen, Di Zhu, Lei Nico Zheng

llms benchmarks

arXiv AI

Jul 31

Diagnosing Fine-Grained Inconsistency Classification in Financial Disclosure Text

arXiv:2607. 26368v1 Announce Type: cross Abstract: Financial disclosures contain numerical claims, temporal statements, entity references, policy commitments, and risk descriptions that may conflict in qualitatively different ways.

By Aman Kumar, Lasitha Vidyaratne, Dipanjan D Ghosh, Arnab Chakrabarti, Ahmed K Farahat

llms rag fine-tuning benchmarks

arXiv AI

Jul 31

A Graph-Native Bitemporal Memory Store for Conversational AI Agents

arXiv:2607. 26520v1 Announce Type: cross Abstract: Conversational AI agents commonly lack persistent memory across sessions.

By Alp Niksarli, Gopesh Baheti

llms rag agents benchmarks

arXiv AI

Jul 31

Shared Symbolic Backbones for Physically Consistent Multi-Output Symbolic Regression

arXiv:2607. 26528v1 Announce Type: cross Abstract: Symbolic regression provides analytical expressions, but it is usually applied one output at a time.

By Manuel Rodriguez

benchmarks

arXiv AI

Jul 31

WhisperRec: Latent Reasoning for Efficient Foundation Recommendation Models

arXiv:2607. 26621v2 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong reasoning capabilities, motivating their adoption as backbones for foundation recommendation models (FRMs).

By Hao Jiang, Peiru Du, Pengfei Yao, Mengting Li, Siyuan Lou, Kuo Cai, Sheng Yu, Qiang Luo, Jian Liang, Ruiming Tang, Fei Pan, Peng Jiang, Wenwu Ou

llms diffusion efficiency benchmarks safety

arXiv AI

Jul 31

FakeIDet3-DB: Refining Digital Attacks and Patch Extraction for Secure ID Benchmarking

arXiv:2607. 26641v1 Announce Type: cross Abstract: Identity document (ID) authentication relies on the structural integrity of complex, high-frequency security patterns.

By Mu\~noz-Haro Javier, Teruel Andres, Tolosana Ruben, DeAlcala Daniel, Vera-Rodriguez Ruben, Morales Aythami, Fierrez Julian

benchmarks safety

arXiv AI

Jul 31

FPSGen: Flexible Point Cloud Scene Generation with BEV-Supported Transport Flows

arXiv:2607. 26645v1 Announce Type: cross Abstract: Existing point-based generative methods for outdoor scenes primarily focus on LiDAR-conditioned completion.

By Wenzhe He, Meng Wang, JiaWei Qian, Jinfeng Xu, Ying Liu, Ruihui Li

efficiency benchmarks safety

arXiv AI

Jul 31

See2Think: Do Multimodal Models Really Use Intermediate Visual States?

arXiv:2607. 26769v1 Announce Type: cross Abstract: Multimodal large language models increasingly use sketches, annotations, tools, and intermediate images during reasoning, but it remains unclear whether they truly rely on these visual states.

By Siyu Yan, Zhuoran Yan, Haiying Xu, Panhao Zhou, Jingyu Chen, Chenhao Ji, Shuo Cao, Yongheng Zhang, Haoze Liu, Siyu Zhang, Xiwen Gu, Yihao Liu, Alex Jinpeng Wang

llms multimodal benchmarks

arXiv AI

Jul 31

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

arXiv:2607. 26791v1 Announce Type: cross Abstract: Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities.

By Lehan Wang, Boli Chen, Ruixue Ding, Pengjun Xie, Jinwei Huang, Zhendong Liu, Shuo Wang, Tao Lei, Xin Ouyang, Xiaomeng Li

llms agents nlp benchmarks

arXiv AI

Jul 31

Progressive Multimodal Alignment for Continual Instruction Tuning

arXiv:2607. 26947v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) rely on a projector to align visual representations with the language embedding space, making it central to cross-modal understanding.

By Duzhen Zhang, Yahan Yu, Qiaoyi Su, Jiahua Dong, Tielin Zhang

llms rag fine-tuning multimodal benchmarks safety

arXiv AI

Jul 31

SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidence

arXiv:2607. 27066v1 Announce Type: cross Abstract: Scientific figure assessment in peer review differs fundamentally from general image quality evaluation: a figure must be visually legible, faithfully support the manuscript's claims, and communicate evidence with a clear visual hierarchy.

By Chuanzhi Xu, Zihan Deng, Huiqi Liang, Chengkun Yue, Zhanlin Cui, Pengfei Ye, Weidong Cai

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jul 31

SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

arXiv:2607. 27084v1 Announce Type: cross Abstract: Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers.

By Zihan Deng, Chuanzhi Xu, Huiqi Liang, Haoyang Li, Xiaozhen Zhong, Lequan Yu

llms agents multimodal benchmarks safety

arXiv AI

Jul 31

MMAC: A Massive Multi-dimensional Benchmark for Audio Captioning

arXiv:2607. 27109v2 Announce Type: cross Abstract: With the development of audio large language models (AudioLLMs), audio captioning needs to move from brief descriptions toward open-ended and fine-grained free-form descriptions.

By Weijie Wu, Junbo Li, Lin Li, Jun Fang, Qingyang Hong

llms benchmarks

arXiv AI

Jul 31

TANDEM: Temporal-Aware Neural Detection for Multimodal Hate Speech

arXiv:2601. 11178v3 Announce Type: replace Abstract: Social media platforms are increasingly dominated by long-form multimodal content, where harmful narratives are constructed through a complex interplay of audio, visual, and textual cues.

By Girish A. Koushik, Helen Treharne, Diptesh Kanojia

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jul 31

FPEdit: Robust LLM Fingerprinting through Localized Parameter Editing

arXiv:2508. 02092v3 Announce Type: replace-cross Abstract: Large language models represent significant investments in computation, data, and engineering expertise, making them extraordinarily valuable intellectual assets.

By Shida Wang, Chaohu Liu, Yubo Wang, Linli Xu

llms fine-tuning efficiency benchmarks safety

arXiv AI

Jul 31

VideoNorms: Benchmarking Cultural Awareness of Video Language Models

arXiv:2510. 08543v2 Announce Type: replace-cross Abstract: As Video Large Language Models (VideoLLMs) are deployed globally, it is important to assess their ability to reason across cultural contexts.

By Nikhil Reddy Varimalla, Yunfei Xu, Meng Fan Wang, Arkadiy Saakyan, Smaranda Muresan

llms benchmarks

Get the daily briefing