Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

18,270 stories · RSS feed

arXiv AI

Jul 3

ADMC: Attention-based Diffusion Model for Missing Modalities Feature Completion

arXiv:2507. 05624v2 Announce Type: replace Abstract: Multimodal emotion and intent recognition is essential for automated human-computer interaction, It aims to analyze users' speech, text, and visual information to predict their emotions or intent.

By Yuhan Li, Wei Zhang, Juan Chen, Jiangjia Yan, Peng Xiangli, Liangze Yin

diffusion multimodal benchmarks

arXiv AI

Jul 3

ESC: Emotional Self-Correction for Reliable Vision-Language Models

arXiv:2607. 02089v1 Announce Type: cross Abstract: Vision-language models (VLMs) have achieved strong performance across diverse multimodal tasks, yet they remain vulnerable to unreliable reasoning.

By Tien-Huy Nguyen, Minh-Nhat Nguyen, Nguyen Nhat Huy, Hung Viet Nguyen, Huy Nguyen Minh Nhat, Thanh-Huy Nguyen, Cuong Tuan Nguyen, Hoang M. Le, Dat Nguyen, Phat Kim Huynh, Min Xu, Ulas Bagci

llms multimodal benchmarks safety

arXiv AI

Jul 3

TUDUM: A Turkish-Thinking Reasoning Pipeline for Qwen3.5-27B

arXiv:2607. 01927v1 Announce Type: cross Abstract: This paper presents TUDUM (T\"urk\c{c}e D\"u\c{s}\"unen \"Uretken Model), a project pipeline for adapting a Qwen-family 27B thinking model toward Turkish reasoning.

By Baran Bingol, Bahaeddin Turkoglu

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 3

AIriskEval-edu: New Dataset for Risk Assessment in AI-mediated K-12 Educational Explanations

arXiv:2607. 01934v1 Announce Type: cross Abstract: This work introduces AIriskEval-edu-db2, a new dataset designed to train and evaluate auditors based on LLMs for an explainable pedagogical risk assessment in instructional content for grades K-12.

By Javier Irigoyen, Roberto Daza, Francisco Jurado, Julian Fierrez, Ruben Tolosana, Alvaro Ortigosa, Enrique Blas, Aythami Morales

llms fine-tuning benchmarks safety

arXiv AI

Jul 3

Assessing VLM Reliability for Medical Image Quality Evaluation Under Corruption and Bias

arXiv:2607. 01973v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) are increasingly applied in medical tasks such as pathology description, report generation, and visual question answering.

By Sofiane Ouaari, Kevin Vorwalder, Nico Pfeifer

llms rag nlp multimodal benchmarks safety

arXiv AI

Jul 3

SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses

arXiv:2607. 02049v1 Announce Type: cross Abstract: Large Language Models are increasingly deployed in emotional-support contexts and crisis-related situations.

By Anna Chorna

llms benchmarks

arXiv AI

Jul 3

eCream-MedCorpus A Large-Scale Corpus of Clinical Notes for Italian

arXiv:2606. 12569v2 Announce Type: replace-cross Abstract: We present eCream-MedCorpus, a new and unique large-scale dataset of clinical notes produced in Emergency Departments of Italian hospitals.

By Tiziano Labruna, Guido Bertolini, Pietro Ferrazzi, Bernardo Magnini

llms benchmarks

arXiv Machine Learning

Jul 3

Fixed-Set Robustness in Programming by Example: Example Corruption and Semantic Partition Recovery

arXiv:2607. 01280v1 Announce Type: new Abstract: Programming-by-example systems infer programs from a small set of input-output examples.

By Yuan Si, Jialu Zhang

llms benchmarks safety

arXiv Machine Learning

Jul 3

I\textsuperscript{2}RiMA: Spectral Riemannian Representation with Temporal Attention for Mental Stress Detection based on EEG Signals

arXiv:2607. 01279v1 Announce Type: new Abstract: Cross-subject EEG stress detection remains challenging because discriminative stress-related patterns are both subject-dependent and frequency-specific.

By Cheng He, Kunyu Peng, Shangen Han, Jinming Ma, Jinhong Ding, Likun Xia

nlp benchmarks

arXiv AI

Jul 3

Hawk: Harnessing Hardware-Aware Knowledge for High-Performance NPU Kernel Generation

arXiv:2607. 01590v1 Announce Type: new Abstract: Developing high-performance kernels for Neural Processing Units (NPUs) is a critical industry bottleneck, requiring developers to manually navigate implicit hardware constraints and strict memory hierarchies.

By Junyi Wen, Ruiyan Zhuang, Yongjia Xu, Pengtu Li, Rui Zou, Hongyi Chen, Chingman Wan, Puxu Yang, Wuhui Chen, Yanlin Wang

llms efficiency benchmarks

arXiv AI

Jul 3

CamoNAS: Neural Architecture Search for Enhanced Camouflaged Object Detection

arXiv:2607. 01870v1 Announce Type: new Abstract: Camouflaged Object Detection (COD) aims to locate and segment objects that blend into their surroundings, presenting challenges due to weak edge cues and ill-defined boundaries.

By Dawei Ren, Yan Zhang, Hongying Tang, Qiaoling Zhou, Jianpo Liu

computer-vision benchmarks

arXiv AI

Jul 3

Path-level Hindsight Instructions for Semantic Exploration in Vision-Language Navigation

arXiv:2607. 01754v1 Announce Type: new Abstract: On-policy exploration is a crucial component for training robust Vision-Language Navigation agents, as it exposes the policy to a broader state distribution.

By Sung June Kim, Sangpil Kim, Honglak Lee

agents robotics multimodal benchmarks

arXiv AI

Jul 3

Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry

arXiv:2607. 01661v1 Announce Type: new Abstract: Multi-agent systems are increasingly used for forecasting future events, as deliberation among multiple LLMs is believed to improve reasoning and calibration.

By Yuante Li, Yicheng Tao, Kate Zhang, Taozhi Wang, Gefei Gu, Yaxin Zhou

llms agents benchmarks

arXiv AI

Jul 3

Generic Expert Coverage for Pruning SparseMixture-of-Experts Language Models

arXiv:2607. 01710v1 Announce Type: new Abstract: Sparsely activated Mixture-of-Experts (MoE) language models contain substantial structured redundancy among routed experts, but pruning them without downstream calibration data remains challenging.

By Yongqin Zeng, Sicheng Pan, Jiale Wang, Hai-tao Zheng, Hong-Gee Kim, Chunxia Ma, XiuTeng Zhou

llms efficiency benchmarks safety

arXiv AI

Jul 3

MKGR: Multimodal Knowledge-Graph Representation Learning for Cold-Start Protein-Protein Interaction Prediction

arXiv:2607. 01627v1 Announce Type: cross Abstract: Accurate protein-protein interaction (PPI) prediction is central to functional genomics, disease mechanism discovery, and drug development.

By Wenbo Zhang

rag multimodal benchmarks

arXiv AI

Jul 3

X-LogSMask: Expand Transformer for Graph-Structured Data

arXiv:2607. 01553v1 Announce Type: cross Abstract: Transformers have become general-purpose architectures, but their all-to-all self-attention is poorly matched to graph data, whose interactions are sparse, structured and multi-scale.

By Leyan Li, Rennong Yang, Zhenxing Zhang, Liping Hu

llms benchmarks safety

arXiv AI

Jul 3

Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning

arXiv:2604. 02091v2 Announce Type: replace-cross Abstract: Rerankers play a pivotal role in refining retrieval results for Retrieval-Augmented Generation.

By Yuhang Wu, Xiangqing Shen, Fanfan Wang, Cangqi Zhou, Zhen Wu, Xinyu Dai, Rui Xia

llms rag reinforcement-learning benchmarks

arXiv Machine Learning

Jul 3

Many Voices, One Reward: Multi-Role Rubric Generation for LLM Judging and Reward Modeling

arXiv:2607. 01830v1 Announce Type: new Abstract: Reliable reward and preference signals are critical for evaluating and optimizing large language models on open-ended tasks.

By Dazhi Fu, Jiuding Yang, Yiwen Guo, Jicong Fan

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jul 3

Frequency Shift Physics-Informed Extreme Learning Machine for Solving High-Frequency Partial Differential Equations

arXiv:2607. 01694v1 Announce Type: new Abstract: Solving partial differential equations (PDEs) with high-frequency solutions remains a central challenge in physics-informed machine learning due to spectral bias -- the tendency of neural networks to learn low-frequency components preferentially.

By Xiong Xiong, Ruonan Zhai, Zheng Zeng, Sheng Zhou, Rongchun Hu, Zichen Deng

diffusion benchmarks safety

arXiv AI

Jul 3

DemoPSD: Disagreement-Modulated Policy Self-Distillation

arXiv:2607. 02502v1 Announce Type: cross Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training large language models (LLMs) to reason, where a single model acts as both the teacher and the student with different levels of information access.

By Yunhe Li, Hao Shi, Wenhao Liu, Mengzhe Ruan, Hanxu Hou, Zhongxiang Dai, Shuang Qiu, Linqi Song

llms efficiency benchmarks

Get the daily briefing