Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

15,813 stories · RSS feed

arXiv AI

Jul 28

Poster: Rethinking Security in LLM Code Generation through Real-World Risk Scenarios

arXiv:2607. 23088v1 Announce Type: cross Abstract: Large Language Models (LLMs) are widely used for code generation, yet their security behavior in realistic development workflows remains underexplored.

By Lixun Ma, Ruolong Ma, Bei Wang, Feng Wei, Zhenguang Liu, Lorenzo Cavallaro, Wentao Chen

llms benchmarks

arXiv AI

Jul 28

False Prophets: On the Security of World Models in Agentic Systems

arXiv:2607. 23147v1 Announce Type: cross Abstract: Large language models now power autonomous agents capable of complex, multi-step tasks in different environments.

By Erik Imgrund, Anna Wimbauer, Klim Kireev, Konrad Rieck

llms agents benchmarks safety

arXiv AI

Jul 28

Token-Region Guided Cross-Attention Fusion for Multimodal Affect Interpretation

arXiv:2607. 23493v1 Announce Type: cross Abstract: Automated analysis of multimodal content on social networks has become a critical task for understanding public sentiment and information diffusion in the digital age.

By Musa Tur Farazi, Nufayer Jahan Reza

llms diffusion computer-vision multimodal benchmarks safety

arXiv AI

Jul 28

AgentOmnia: Scaling Agentic Models for Full-Scenario Applications

arXiv:2607. 23124v1 Announce Type: new Abstract: Large language model agents have advanced rapidly, yet progress remains fragmented across domains, capabilities, task difficulty, and interaction settings.

By Hao Jiang, Gangtao Xin, Yingdi Huang, Guojie Zhu, Jiangshan Zhang, Xinyuan Lin, Yunkun Xu, Chengyu Shen, Wenlong Fei, Jiawei Li, Yujie Fu, Sichen Kang, Tingyu Xie, Yedi Hu, Jingren Zhang, Hongcheng Gao, Jianshu Zeng, Chong Chen, Chang Guo, Chao Feng, Feng Wang, Fulin Lin, Jinchao Ma, Lang Mei, Li Huang, Liyan Liu, Qing He, Shuting Tao, Siyu Mo, Xiangnan Chen, Xiaohan Yu, Xiaoyang Li, Yanheng Hou, Yanyu Wu, Zhihan Yang, Wentao Zhang, Yang Gao, Zhao Cao

llms agents reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 28

AI-generated Images Challenge Visual Trust in High-risk Scenarios

arXiv:2607. 22745v1 Announce Type: cross Abstract: Rapid advances in image generation are eroding the evidentiary value of visual content in settings where authenticity can affect public safety and personal reputation.

By Yi-Zhi Wang, Yichen Xiao, Linan Yue, Weibo Gao, Yichao Du, Pengfei Fang, Shimin Di, Min-Ling Zhang

llms diffusion multimodal benchmarks safety

arXiv AI

Jul 28

Generative Artificial Intelligence (GenAI) to convert images of queuing networks into verifiable simulation models: an open-weight LLM workflow approach

arXiv:2607. 24259v1 Announce Type: new Abstract: Recent work has explored the use of Large Language Models (LLMs) to automate simulation model building, typically by generating executable code directly from natural language descriptions.

By Thomas Monks, Alison Harper, Amy Heather, Navonil Mustafee

llms multimodal benchmarks

arXiv AI

Jul 28

Multinex: Lightweight Low-light Image Enhancement via Multi-prior Retinex

arXiv:2604. 10359v3 Announce Type: replace-cross Abstract: Low-light image enhancement (LLIE) aims to restore natural visibility, color fidelity, and structural detail under severe illumination degradation.

By Alexandru Brateanu, Tingting Mu, Codruta Ancuti, Cosmin Ancuti

efficiency benchmarks

arXiv AI

Jul 28

DOSA: A Tree-Guided, Self-Regressive Framework for Long Document Structure Analysis

arXiv:2607. 22679v1 Announce Type: new Abstract: In visually-rich documents, information is encoded not only in individual page objects such as tables, headers, and text blocks, but also in the structural relations among them, making document structure analysis fundamental to information retrieval and document understanding.

By Bohou Li, Benjamin Sowell, Mehul Shah, Mark Lindblad, Henry Lindeman

benchmarks

arXiv AI

Jul 28

Design Theater: A Benchmark for Generative UI

arXiv:2607. 22928v1 Announce Type: new Abstract: Generative UI tools promise to democratize UI design by turning natural language descriptions into complete interfaces.

By Kashif Imteyaz, Kaif Imteyaz, Nakul Rajpal, Kaif Shaikh, Michael Muller, Saiph Savage

benchmarks

arXiv AI

Jul 28

ConsistencyGate: Preventing Memory Contamination in LLM Agents via Self-Consistency Admission Control

arXiv:2607. 22962v1 Announce Type: new Abstract: LLM agents that operate over many turns accumulate facts in an external memory store and reuse them as premises for downstream reasoning.

By Yan Zhang, Shibo Li

llms agents fine-tuning benchmarks

arXiv AI

Jul 28

Efficient LLM-Generated Shuttling Compilers for Complex Trapped-Ion Architectures

arXiv:2607. 24714v1 Announce Type: cross Abstract: Trapped-ion quantum computers rely on shuttling compilers, which cast an input algorithm into a sequence of ion-qubit movements within a given architecture.

By Fabian Kreppel, Reza Salkhordeh, Ferdinand Schmidt-Kaler, Andr\'e Brinkmann

llms benchmarks

arXiv AI

Jul 28

ObsDriveBench: Benchmarking Multimodal Understanding under Adverse Weather with Observability Awareness

arXiv:2607. 23537v1 Announce Type: new Abstract: Autonomous driving under adverse weather remains a critical challenge, yet existing vision-language benchmarks mainly evaluate under standard conditions, synthetic corruptions, or single modality.

By Qiao Yan, Yihan Wang, Zhenghao Xing, Jiaqi Xu, Pheng-Ann Heng

llms agents reinforcement-learning robotics fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 28

Music-Source-Separation-Training (MSST): A Unified Framework for Training and Evaluating Music Demixing Models

arXiv:2607. 23395v1 Announce Type: cross Abstract: Music Source Separation (MSS), the task of recovering individual sound components (stems) from a polyphonic mixture, is central to applications ranging from karaoke and remixing to audio restoration and content production.

By Roman Solovyev, Ilya Kiselev, Alexander Stempkovskiy, Tatiana Gabruseva

fine-tuning benchmarks

arXiv AI

Jul 28

Co-Learning for Missing Arbitrary Modalities in Multi-modal Classification

arXiv:2607. 24683v1 Announce Type: cross Abstract: Multi-modal classification leverages complementary information across diverse data sources to enhance predictive performance.

By Francisco Mena, Dino Ienco, Roberto Interdonato, Cassio F. Dantas, Simon Besnard

multimodal benchmarks safety

arXiv AI

Jul 28

Infinite-Precision Autoregressive Modeling for Vector Graphics and Layouts

arXiv:2601. 05680v2 Announce Type: replace-cross Abstract: While Transformer-based autoregressive models excel in data generation, their token discretization strategy inherently limits their precision in continuous domains.

By Yeonsang Shin, Insoo Kim, Bongkeun Kim, Keonwoo Bae, Bohyung Han

llms diffusion benchmarks

arXiv AI

Jul 28

How Transformers Learn to Plan via Multi-Token Prediction

arXiv:2604. 11912v2 Announce Type: replace-cross Abstract: While next-token prediction (NTP) has been the standard objective for training language models, it often struggles to capture global structure in reasoning tasks.

By Jianhao Huang, Zhanpeng Zhou, Renqiu Xia, Baharan Mirzasoleiman, Weijie Su, Wei Huang

llms benchmarks safety

arXiv AI

Jul 28

TextRich: A Multi-Domain Benchmark for Detecting AI-Generated Text-Rich Images from GPT-Image-2

arXiv:2606. 19259v2 Announce Type: replace-cross Abstract: Text-rich images often contain privacy-sensitive, transactional, or decision-relevant information.

By Yijin Wang, Shuyi Wang, Wenhan Zhang, Yuqi Ouyang

llms diffusion multimodal benchmarks safety

arXiv AI

Jul 28

STAIF: A Stage-wise Optimization for Complex Instruction Following

arXiv:2607. 22649v1 Announce Type: new Abstract: Following complex instructions with multiple explicit constraints remains a fundamental challenge for large language models (LLMs).

By Jian Hong, Chen Cheng, Quan Liu, Yuhao Chen, Enhong Chen

llms reinforcement-learning benchmarks safety

arXiv AI

Jul 28

Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifiers

arXiv:2509. 03059v2 Announce Type: replace-cross Abstract: Recent advances in Large Language Models (LLMs) have shown that their reasoning capabilities can be significantly improved through Reinforcement Learning with Verifiable Reward (RLVR), particularly in domains like mathematics and programming, where ground-truth correctness can be automatically evaluated.

By Xingyue Huang, Rishabh, Gregor Franke, Ziyi Yang, Jiamu Bai, Weijie Bai, Jinhe Bi, Zifeng Ding, Yiqun Duan, Chengyu Fan, Wendong Fan, Xin Gao, Ruohao Guo, Yuan He, Zhuangzhuang He, Xianglong Hu, Neil Johnson, Bowen Li, Fangru Lin, Siyu Lin, Tong Liu, Yunpu Ma, Hao Shen, Hao Sun, Beibei Wang, Fangyijie Wang, Hao Wang, Haoran Wang, Yang Wang, Yifeng Wang, Zhaowei Wang, Ziyang Wang, Yifan Wu, Zikai Xiao, Chengxing Xie, Fan Yang, Junxiao Yang, Qianshuo Ye, Ziyu Ye, Guangtao Zeng, Yuwen Ebony Zhang, Zeyu Zhang, Zihao Zhu, Bernard Ghanem, Philip Torr, Guohao Li

llms agents reinforcement-learning benchmarks

arXiv AI

Jul 28

TokenMem: Faithful Knowledge Injection for Frozen LLMs

arXiv:2607. 22625v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge, but suffers from knowledge conflicts: when retrieved information contradicts parametric memory, the shared self-attention pathway produces unpredictable outputs.

By Chengzhang Yu, Chenyang Zheng, Zening Lu, Yingru He, Yutong Huang, Yiming Zhang, Yue Xu, Zhanpeng Jin

llms rag benchmarks

Get the daily briefing