Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,245 stories · RSS feed

arXiv AI

Jun 9

Capacity-Controlled Global Attention for Graph Transformers

arXiv:2604. 17324v2 Announce Type: replace-cross Abstract: Global self-attention drives modern graph transformers, yet the softmax at its core imposes a structural constraint rarely examined directly: every attention row is non-negative and sums to one, so each per-head output is a mass-conserving convex combination of value vectors.

By Yang Liu, Dongxin Guo, Tom Zheng, Siu Ming Yiu, Liam Ning, Jikun Wu

llms benchmarks

arXiv Machine Learning

Jun 9

LOTTERY: Learning from Reference-Only Samples in Two-Sample Testing under Size Asymmetry

arXiv:2606. 08460v1 Announce Type: cross Abstract: Data-adaptive two-sample testing assesses if two samples come from the same distribution, using a discrepancy learned from the data (e.

By Xunye Tian, Zhijian Zhou, Liuhua Peng, Feng Liu

benchmarks

arXiv AI

Jun 9

Cosmo3DFlow: Wavelet Flow Matching for Spatial-to-Spectral Compression in Reconstructing the Early Universe

arXiv:2602. 10172v2 Announce Type: replace-cross Abstract: Reconstructing the early universe from the evolved present-day universe is a challenging and computationally demanding problem in modern astrophysics.

By Md. Khairul Islam, Zeyu Xia, Ryan Goudjil, Jialu Wang, Arya Farahi, Judy Fox

diffusion efficiency benchmarks

arXiv Machine Learning

Jun 9

Multi-View Speech Representation Learning for Parkinson's Disease Detection Using Context-guided Cross-modal Attention

arXiv:2606. 09271v1 Announce Type: cross Abstract: Parkinson's disease (PD) is a progressive neurodegenerative disorder that frequently causes speech impairments associated with hypokinetic dysarthria.

By George Theodosiou, Loukas Ilias, Dimitris Askounis

rag multimodal benchmarks

arXiv Machine Learning

Jun 9

Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation

arXiv:2606. 08679v1 Announce Type: cross Abstract: Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts.

By Bitya Neuhof, Yuval Benjamini

benchmarks

arXiv AI

Jun 9

ACTIVE-o3: Empowering MLLMs with Active Perception via Pure Reinforcement Learning

arXiv:2505. 21457v2 Announce Type: replace-cross Abstract: Active vision, also known as active perception, refers to actively selecting where and how to look in order to gather task-relevant information.

By Muzhi Zhu, Hao Zhong, Canyu Zhao, Zongze Du, Mingyu Liu, Zheng Huang, Anzhou Li, Hao Chen, Cheng Zou, Jingdong Chen, Ming Yang, Chunhua Shen

llms agents computer-vision reinforcement-learning robotics multimodal benchmarks

arXiv AI

Jun 9

Brain2Text Decoding Model Reveals the Neural Mechanisms of Visual Semantic Processing

arXiv:2503. 22697v3 Announce Type: replace-cross Abstract: Decoding sensory experiences from neural activity to reconstruct human-perceived visual stimuli and semantic content remains a challenge in neuroscience and artificial intelligence.

By Feihan Feng, Jingxin Nie

llms benchmarks

arXiv AI

Jun 9

XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad

arXiv:2601. 14063v2 Announce Type: replace-cross Abstract: Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cultural contexts.

By Mohsinul Kabir, Tasnim Ahmed, Md Mezbaur Rahman, Shaoxiong Ji, Hassan Alhuzali, Yuechen Jiang, Jimin Huang, Sophia Ananiadou

llms nlp benchmarks

arXiv AI

Jun 9

DecepGPT: Schema-Driven Deception Detection with Multicultural Datasets and Robust Multimodal Learning

arXiv:2603. 23916v3 Announce Type: replace-cross Abstract: Multimodal deception detection aims to identify deceptive behavior by analyzing audiovisual cues for forensics and security.

By Jiajian Huang, Dongliang Zhu, Zitong YU, Hui Ma, Jiayu Zhang, Chunmei Zhu, Xiaochun Cao

efficiency multimodal benchmarks

arXiv AI

Jun 9

MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering

arXiv:2601. 22859v3 Announce Type: replace-cross Abstract: The evolution of Large Language Model (LLM) agents for software engineering (SWE) is constrained by the scarcity of verifiable datasets, a bottleneck stemming from the complexity of constructing executable environments across diverse languages.

By Chuanzhe Guo, Jingjing Wu, Sijun He, Yang Chen, Zhaoqi Kuang, Shilong Fan, Bingjin Chen, Siqi Bao, Jing Liu, Hua Wu, Qingfu Zhu, Wanxiang Che, Haifeng Wang

llms agents benchmarks

arXiv Machine Learning

Jun 9

Causal Semantic Alignment for LLM-based Time Series Forecasting

arXiv:2606. 08262v1 Announce Type: new Abstract: Recent advances in Large Language Models (LLMs) have opened new possibilities for time series forecasting by enabling alignment between temporal patterns and pretrained word embeddings.

By Kexuan Zhang, Xiaobei Zou, Cesare Alippi, Gary G. Yen, Yang Tang

llms rag benchmarks safety

arXiv Machine Learning

Jun 9

Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

arXiv:2606. 09456v1 Announce Type: new Abstract: On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models.

By Yifan Niu, Han Xiao, Dongyi Liu, Zelong Wang, Dihong Gong, Yasheng Wang, Jia Li

llms nlp fine-tuning efficiency benchmarks

arXiv AI

Jun 9

CoVEBench: Can Video Editing Models Handle Complex Instructions?

arXiv:2606. 08415v1 Announce Type: cross Abstract: While recent text-guided video editing models excel at elementary tasks (e.

By Jiangtao Wu, Jiaming Wang, Yiwen He, Yuanxing Zhang, Shihao Li, Dunyuan Liu, Xuedong Zhao, Jialu Chen, Zekun Moore Wang, Jiaheng Liu

benchmarks

arXiv AI

Jun 9

VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generation

arXiv:2601. 23286v4 Announce Type: replace-cross Abstract: While recent video diffusion models (VDMs) produce visually impressive results, they fundamentally struggle to maintain 3D structural consistency, often resulting in object deformation or spatial drift.

By Hongyang Du, Junjie Ye, Xiaoyan Cong, Runhao Li, Jingcheng Ni, Aman Agarwal, Zeqi Zhou, Zekun Li, Randall Balestriero, Yue Wang

diffusion reinforcement-learning benchmarks safety

arXiv AI

Jun 9

Condition-Gated Reasoning for Context-Dependent Biomedical Question Answering

arXiv:2602. 17911v3 Announce Type: replace-cross Abstract: Current biomedical question answering (QA) systems often assume that medical knowledge applies uniformly, yet real-world clinical reasoning is inherently conditional: nearly every decision depends on patient-specific factors such as comorbidities and contraindications.

By Jash Rajesh Parekh, Wonbin Kweon, Joey Chan, Rezarta Islamaj, Robert Leaman, Pengcheng Jiang, Chih-Hsuan Wei, Zhizheng Wang, Zhiyong Lu, Jiawei Han

rag nlp benchmarks

arXiv AI

Jun 9

Vision Hopfield Memory Networks for Image Recognition

arXiv:2603. 25157v3 Announce Type: replace-cross Abstract: Recent vision backbones, such as Transformer families and state-space models like Mamba, have achieved remarkable progress on image recognition.

By Jianfeng Wang, Amine M'Charrak, Luk Koska, Xiangtao Wang, Daniel Petriceanu, Ruizhi Wang, Michael Bumbar, Luca Pinchetti, Thomas Lukasiewicz

llms computer-vision benchmarks safety

arXiv AI

Jun 9

Shared Latent Structures Enable Unified Backdoor Detection and Mitigation in LLMs

arXiv:2606. 07963v1 Announce Type: new Abstract: Backdoor attacks in large language models (LLMs) are often treated as isolated trigger-response failures, motivating defenses tailored to specific triggers or behaviors.

By Omar Mahmoud, Aly M. Kassem, Thommen George Karimpanal, Buddhika Laknath Semage, Negar Rostamzadeh, Golnoosh Farnadi, Santu Rana

llms robotics fine-tuning benchmarks safety

arXiv Machine Learning

Jun 9

Operator learning for the 2D incompressible Navier-Stokes equations: a conformal prediction approach in the data-scarce regime

arXiv:2606. 08654v1 Announce Type: new Abstract: In this paper, we propose a perturbation-based conformal prediction framework for uncertainty quantification in operator learning, with a focus on the 2D Navier--Stokes equations.

By Weinan Wang, Bowen Gang, Hao Deng

benchmarks

arXiv Machine Learning

Jun 9

Beyond Convolution: Advancing Hypergraph Neural Networks with Hypergraph U-Nets

arXiv:2606. 09051v1 Announce Type: new Abstract: Convolutions have successfully transitioned from image processing to the complex realm of non-Euclidean higher-order domains, particularly in hypergraphs.

By Fuli Wang, Wei Qian, Daniel L. Lau, Gonzalo R. Arce

benchmarks

arXiv AI

Jun 9

See More, Think Deeper: Query-Expanded Visual Evidence and Answer-Clue Guided Reflection for Long Video Understanding

arXiv:2606. 09064v1 Announce Type: cross Abstract: Recent advances in Video Large Language Models (Video-LLMs) have enabled performance on long-video understanding tasks.

By Shuning Wang, Zhiheng Wu, YiNuo Lu, Naiming Liu, Chen Jia, Bowen Liu, Shuo Nie, Weijie Zhu, Yumeng Zhang

llms benchmarks

Get the daily briefing