Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,071 stories · RSS feed

arXiv Machine Learning

Jun 19

Multimodal Concept Bottleneck Models

arXiv:2606. 19882v1 Announce Type: cross Abstract: Concept Bottleneck Models (CBMs) enhance the interpretability of deep learning networks by aligning the features extracted from images with natural concepts.

By Tongqing Shi, Ge Yan, Tuomas Oikarinen, Tsui-Wei Weng

rag multimodal benchmarks safety

arXiv AI

Jun 19

PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models

arXiv:2606. 19534v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have achieved remarkable progress in visual understanding tasks.

By Yueyi Sun, Yuhao Wang, Jason Li, Ye Tian, Tao Zhang, Jacky Mai, Yihan Wang, Haochen Wang, Jinbin Bai, Ling Yang, Yunhai Tong

llms diffusion multimodal benchmarks

arXiv AI

Jun 19

MakeupMirror: Improving Facial Attribute Preservation in Diffusion Models for Makeup Transfer

arXiv:2606. 20094v1 Announce Type: cross Abstract: Makeup transfer models enable fun augmented reality (AR) experiences as well as virtual try-on (VTO) for online makeup shopping.

By Nefeli Andreou, Angel Mart\'inez-Gonz\'alez, Sabine Sternig, Matthieu Guillaumin, Epameinondas Antonakos, Michael Opitz

diffusion benchmarks

arXiv AI

Jun 19

CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models

arXiv:2606. 19788v1 Announce Type: new Abstract: We present CombEval, a dynamic benchmark for evaluating combinatorial counting in large language models.

By Yuxu Zhou, Ond\v{r}ej Ku\v{z}elka, Yuyi Wang, Yuanhong Wang, Yi Chang

llms benchmarks

arXiv AI

Jun 19

Implicit Semantic-Aware Communication Based on Hypergraph Reasoning

arXiv:2606. 20162v1 Announce Type: new Abstract: Semantic-aware communication has emerged as a transformative paradigm for next-generation communication systems, shifting the fundamental goal from transmitting bit-level symbols to reliably recovering and understanding the semantic meaning of information.

By Yiwei Liao, Shurui Tu, Yong Xiao, Yingyu Li, Guangming Shi

rag benchmarks

arXiv AI

Jun 19

Evaluating and Enhancing Negation Comprehension in Remote Sensing MLLMs

arXiv:2606. 20177v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have demonstrated remarkable success in various Remote Sensing (RS) tasks.

By Haochen Han, Jue Wang, Alex Jinpeng Wang, Fangming Liu

llms multimodal benchmarks safety

arXiv AI

Jun 19

Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models

arXiv:2510. 21978v2 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) has delivered impressive gains in mathematical and multimodal reasoning and has become a standard post-training paradigm for contemporary language and vision-language models.

By Hoang Phan, Xianjun Yang, Yuanshun Yao, Jingyu Zhang, Shengjie Bi, Xiaocheng Tang, Madian Khabsa, Lijuan Liu, Deren Lei

llms reinforcement-learning multimodal benchmarks

arXiv Machine Learning

Jun 19

Prompt, Plan, Extract: Zero-Shot Agentic LLMs Workflows for Lung Pathology Extraction from Clinical Narratives

arXiv:2606. 19852v1 Announce Type: cross Abstract: Information extraction from pathology reports is essential for cancer staging, tumor registry population.

By Aman Pathak, Cheng Peng, Mengxian Lyu, Ziyi Chen, Reema Solan, Sankalp Talankar, Yasir Khan, Hiren Mehta, Aokun Chen, Yi Guo, Yonghui Wu

llms agents nlp benchmarks

arXiv AI

Jun 19

Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inference

arXiv:2606. 20245v1 Announce Type: new Abstract: Large language models (LLMs) have achieved strong performance across a wide range of language-based tasks by leveraging both extensive parametric knowledge and in-context learning ability, enabling them to incorporate external information provided in the input prompt.

By Huang Peng, Jiuyang Tang, Weixin Zeng, Hao Xu, Xiang Zhao

llms agents benchmarks

arXiv AI

Jun 19

Lagrange: An Open-Vocabulary, Energy-Based Sparse Framework for Generalized End-to-End Driving

arXiv:2606. 20274v1 Announce Type: new Abstract: Scaling end-to-end autonomous driving to complex, open-world environments requires perceptual models that generalize to anomalous scenarios and planners that produce kinematically valid trajectories.

By Shihao Ji, HongXi Li, Zihui Song, Mingyu Li

llms agents robotics multimodal benchmarks

arXiv AI

Jun 19

Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

arXiv:2606. 20381v1 Announce Type: new Abstract: FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements.

By Qian Zhao, Kunlong Chen, Changxin Tian, Zhonghui Jiang, Haitao Zhang, Chaofan Yu, Peijie Jiang, Mingliang Gong, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou

llms efficiency benchmarks safety

arXiv AI

Jun 19

FlowEdit: Associative Memory for Lifelong Pronunciation Adaptation in Flow-Matching TTS

arXiv:2606. 20518v1 Announce Type: new Abstract: Flow-matching text-to-speech systems achieve remarkable zero-shot quality but remain static after deployment: pronunciation errors on out-of-vocabulary proper nouns persist unless the model is retrained.

By Harshit Singh, Ayush Pratap Singh, Nityanand Mathur

rag multimodal benchmarks

arXiv Machine Learning

Jun 19

PaAno+: Multiscale Encoding and Cross-Variable Attention for Time Series Anomaly Detection

arXiv:2606. 20055v1 Announce Type: new Abstract: Time-series anomaly detection has significant practical value for industrial and medical monitoring, as well as other critical domains.

By Youji Zhu, Hongbing Wang, Wenchao Liu, Xiaodong Liu, Xiangguang Xiong

llms rag benchmarks

arXiv Machine Learning

Jun 19

Semantic-Anchored Evidential Fusion for Domain-Robust Whole-Slide Survival Analysis

arXiv:2606. 19966v1 Announce Type: cross Abstract: Whole-slide images (WSIs) are widely used for computational cancer prognosis.

By Yucheng Xing, Ling Huang, Pei Liu, Jingying Ma, Jiaqing Xu, Kai He, Mengling Feng

nlp benchmarks

arXiv Machine Learning

Jun 19

GEMS: Geometric Constraints Enable Multi-Semantic Superposition in LLMs

arXiv:2606. 19946v1 Announce Type: cross Abstract: Activation steering controls model behavior by modifying intermediate hidden states at inference time without retraining.

By Yu Deng

llms benchmarks

arXiv AI

Jun 19

Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

arXiv:2606. 20517v1 Announce Type: new Abstract: LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks.

By Maria Ivanova, Pavel Zadorozhny, Rodion Levichev, Ivan Petrov, Adamenko Pavel, Ivan Lopatin, Alexey Kutalev, Dmitrii Babaev

llms benchmarks

arXiv AI

Jun 19

MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learning

arXiv:2506. 14990v3 Announce Type: replace Abstract: Benchmarks play a central role in reinforcement learning (RL) research, yet their computational constraints often shape what is studied.

By Tristan Tomilin, Luka van den Boogaard, Samuel Garcin, Constantin Ruhdorfer, Bram Grooten, Fabrice Kusters, Yali Du, Andreas Bulling, Mykola Pechenizkiy, Meng Fang

agents reinforcement-learning benchmarks

arXiv AI

Jun 19

CREDENCE: Claim Reduction for Decomposition & Enhanced Credibility -- Semantic Metrics and Convergence Analysis

arXiv:2606. 19819v1 Announce Type: cross Abstract: Decomposing compound sentences into atomic, verifiable claims is a prerequisite for reliable automated fact-checking.

By Phuong Huu Vu Tran, Thuan Duc Mai, Bach Xuan Le

llms benchmarks

arXiv AI

Jun 19

UniMM: A Unified Mixture Model Framework for Multi-Agent Simulation

arXiv:2501. 17015v2 Announce Type: replace Abstract: Simulation plays a crucial role in assessing autonomous driving systems, where the generation of realistic multi-agent behaviors is a key aspect.

By Longzhong Lin, Xuewu Lin, Kechun Xu, Haojian Lu, Lichao Huang, Rong Xiong, Yue Wang

agents robotics multimodal benchmarks safety

arXiv AI

Jun 19

Which Pairs to Compare for LLM Post-Training?

arXiv:2606. 19607v1 Announce Type: new Abstract: Preference-based post-training has become a central paradigm for aligning language models.

By Jiangze Han, Vineet Goyal, Will Ma

llms reinforcement-learning benchmarks

Get the daily briefing