Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

23,288 stories · RSS feed

arXiv AI

Jun 6

Benchmarking Emergent Coordination in Large-Scale LLM Populations: An Evaluation Framework on the MoltBook Archive

arXiv:2603. 03555v3 Announce Type: replace-cross Abstract: As multi-agent Large Language Model (LLM) systems scale, evaluating their emergent coordination dynamics becomes increasingly critical.

By Brandon Yee, Pairie Koh

llms agents diffusion benchmarks

arXiv AI

Jun 6

IDEAL: Leveraging Infinite and Dynamic Characterizations of Large Language Models for Query-focused Summarization

arXiv:2407. 10486v3 Announce Type: replace Abstract: Query-focused summarization (QFS) aims to produce summaries that answer particular questions of interest, enabling greater user control and personalization.

By Jie Cao, Dian Jiao, Yang Dai, Rolan Yan, Wenqiao Zhang, Siliang Tang

llms nlp benchmarks safety

arXiv AI

Jun 6

World-Language-Action Model for Unified World Modeling, Language Reasoning, and Action Synthesis

arXiv:2606. 05979v1 Announce Type: cross Abstract: We propose world-language-action (WLA) models as a new class of embodied foundation models.

By Yi Yang, Zhihong Liu, Siqi Kou, Yiyang Chen, Yanzhe Hu, Jianbo Zhou, Boyuan Zhao, Zhijie Wei, Xiao Xia, Xueqi Li, Pengfei Liu, Zhijie Deng

llms diffusion robotics multimodal benchmarks

arXiv AI

Jun 6

Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows

arXiv:2606. 05670v1 Announce Type: new Abstract: Does adding more agents help an LLM workflow once compared systems share the same benchmark loader, tool access, answer contract, usage accounting, and trajectory logging?

By Yuhang Fu, Ruishan Fang, Jiaqi Shao, Huiyu Zheng, Zhengtao Zhu, Bing Luo, Tao Lin

llms agents benchmarks

arXiv AI

Jun 6

Toward Culturally Aligned LLMs through Ontology-Guided Multi-Agent Reasoning

arXiv:2601. 21700v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) increasingly support culturally sensitive decision making, yet often exhibit misalignment due to skewed pretraining data and the absence of structured value representations.

By Wonduk Seo, Wonseok Choi, Junseo Koh, Juhyeon Lee, Hyunjin An, Minhyeong Yu, Jian Park, Qingshan Zhou, Seunghyun Lee, Yi Bu

llms agents benchmarks safety

arXiv AI

Jun 6

SubtleMemory: A Benchmark for Fine-Grained Relational Memory Discrimination in Long-Horizon AI Agents

arXiv:2606. 05761v1 Announce Type: new Abstract: Persistent AI assistants, such as OpenClaw, accumulate large collections of related memories over long-term interactions.

By Wenxuan Wang, Haoyu Sun, Fukuan Hou, Mingyang Song, Weinan Zhang, Yu Cheng, Yang Yang

agents benchmarks

arXiv AI

Jun 6

Benchmarking Open-Source Layout Detection Models for Data Snapshot Extraction from Institutional Documents

arXiv:2606. 06242v1 Announce Type: cross Abstract: Institutional documents contain substantial amounts of operational and analytical information embedded within figures and tables.

By AJ Carl P. Dy, Aivin V. Solatorio

benchmarks

arXiv AI

Jun 6

HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decoupling

arXiv:2510. 00054v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have made significant strides in visual understanding tasks.

By Xianjie Liu, Yiman Hu, Yixiong Zou, Liang Wu, Jian Xu, Bo Zheng

llms reinforcement-learning multimodal benchmarks safety

arXiv AI

Jun 6

PC-Talk: Precise Facial Animation Control for Audio-Driven Talking Face Generation

arXiv:2503. 14295v3 Announce Type: replace-cross Abstract: Recent advancements in audio-driven talking face generation have made great progress in lip synchronization.

By Baiqin Wang, Xiangyu Zhu, Fan Shen, Hao Xu, Zhen Lei

benchmarks safety

arXiv AI

Jun 6

Learning Visual Spatial Planning from Symbolic State via Modality-Gap-Aware Self-Distillation

arXiv:2606. 06076v1 Announce Type: new Abstract: While vision-language models excel at general multimodal understanding, they still struggle with visual spatial planning.

By Haocheng Luo, Jiahui Liu, Ruicheng Zhang, Zhizhou Zhong, Jiaqi Huang, Zunnan Xu, Quan Shi, Jun Zhou, Xiu Li

llms efficiency multimodal benchmarks

arXiv AI

Jun 6

Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation

arXiv:2605. 15913v4 Announce Type: replace-cross Abstract: Block attention, which processes the input as separate blocks that cannot attend to one another, offers significant potential to improve KV cache reuse in long-context scenarios such as Retrieval-Augmented Generation (RAG).

By Shuaiyi Li, Zhisong Zhang, Yan Wang, Lei Zhu, Dongyang Ma, Chenlong Deng, Yang Deng, Wai Lam

rag computer-vision fine-tuning efficiency benchmarks

arXiv AI

Jun 6

Edit-R2: Context-Aware Reinforcement Learning for Multi-Turn Image Editing

arXiv:2606. 05950v1 Announce Type: new Abstract: Text-guided image editing has advanced rapidly with diffusion models and unified multimodal foundation models.

By Yuxiao Ye, Haoran He, Fangyuan Kong, Xintao Wang, Pengfei Wan, Kun Gai, Ling Pan

diffusion reinforcement-learning multimodal benchmarks

arXiv AI

Jun 6

Dense Contexts Are Hard Contexts: Lexical Density Limits Effective Context in LLMs

arXiv:2606. 06203v1 Announce Type: cross Abstract: Input length and the position of relevant information are widely cited as the primary causes of degraded LLM long-context performance.

By Giovanni Dettori, Matteo Boffa, Danilo Giordano, Idilio Drago, Marco Mellia

llms benchmarks

arXiv AI

Jun 6

Assessing the Geographic Diversity of AI's Platial Representations in Image Generation

arXiv:2606. 05188v1 Announce Type: cross Abstract: (Gen)AI diversity is not merely an ethical issue.

By Zilong Liu, Krzysztof Janowicz, Mina Karimi

llms diffusion multimodal benchmarks safety

arXiv AI

Jun 6

Towards One-to-Many Temporal Grounding

arXiv:2606. 06294v1 Announce Type: cross Abstract: Temporal Grounding (TG) aims to localize video segments corresponding to a textual query.

By Qi Xu, Yue Tan, Shihao Chen, Jiahao Meng, Anna Wang, Shunping Ji, Hao Fei, Jason Li

llms benchmarks

arXiv AI

Jun 6

Adapting Diffusion Language Models for Lossless Pixel-Level Image Transmission

arXiv:2606. 06273v1 Announce Type: cross Abstract: Lossless pixel-level image transmission is a fundamental regime beyond semantic communications, because exact recovery requires both accurate symbol probability modeling and reliable delivery over noisy channels.

By Tianqi Ren, Rongpeng Li, Xianfu Chen, Yingyu Li, Zhifeng Zhao

llms diffusion benchmarks

arXiv AI

Jun 6

Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs

arXiv:2606. 05966v1 Announce Type: cross Abstract: Understanding and reasoning about the physical world is the foundation of intelligent behavior, yet state-of-the-art vision-language models (VLMs) still fail at causal physical reasoning, often producing plausible but incorrect answers.

By Tianyi Tang, Zhuoyi Lin, Zeyu Feng, Tianyi Ma, Yew-Soon Ong, Ivor Tsang, Haiyan Yin

llms fine-tuning multimodal benchmarks safety

arXiv AI

Jun 6

Harnessing Structural Context for Entity Alignment Foundation Models

arXiv:2606. 06109v1 Announce Type: cross Abstract: Entity alignment (EA) aims to identify equivalent entities across heterogeneous knowledge graphs (KGs) and is a key component of knowledge fusion and cross-KG reasoning.

By Xingyu Chen, Yuanning Cui, Zequn Sun, Wei Hu

benchmarks safety

arXiv AI

Jun 6

Multi-Granularity Reasoning for Natural Language Inference

arXiv:2606. 05181v1 Announce Type: cross Abstract: Natural Language Inference (NLI) is a fundamental task in natural language understanding that requires determining the logical relationship between a premise and a hypothesis.

By Chunling Xi, Di Liang

llms benchmarks

arXiv AI

Jun 6

LoRi: Low-Rank Distillation for Implicit Reasoning

arXiv:2606. 05315v1 Announce Type: cross Abstract: Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting.

By Ryan Solgi, Jiayi Tian, Zheng Zhang

llms efficiency benchmarks

Get the daily briefing