Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

24,507 stories · RSS feed

arXiv Machine Learning

Jun 2

Design-MLLM: A Reinforcement Alignment Framework for Verifiable and Aesthetic Interior Design

arXiv:2603. 13312v2 Announce Type: replace-cross Abstract: Interior design is a requirements-to-visual-plan generation process that must simultaneously satisfy verifiable spatial feasibility and comparative aesthetic preferences.

By Yuxuan Yang, Xiaotong Mao, Jingyao Wang, Fuchun Sun

llms multimodal benchmarks safety

arXiv AI

Jun 2

Learning When to Translate for Multilingual Reasoning

arXiv:2606. 02465v1 Announce Type: cross Abstract: Reasoning language models (RLMs) achieve strong performance on complex reasoning tasks, but still exhibit substantial multilingual reasoning gaps, largely due to language-understanding failures in non-English inputs.

By Deokhyung Kang, Hyounghun Kim, Gary Geunbae Lee

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jun 2

GRASP: Plan-Guided Graph Retrieval with Adaptive Fusion and Reranking on Semi-Structured Knowledge Bases

arXiv:2605. 30237v2 Announce Type: replace-cross Abstract: Semi-structured knowledge bases (SKBs) embed textual documents in a typed graph of entities and relations, and underpin applications such as product search, academic paper search, and precision-medicine inquiries.

By Yicheng Tao, Yiqun Wang, Xiangchen Song, Xin Luo, Kai Liu, Jie Liu

rag fine-tuning benchmarks

arXiv AI

Jun 2

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution

arXiv:2603. 05308v3 Announce Type: replace-cross Abstract: Assessing whether an article supports an assertion is essential for hallucination detection and claim verification.

By Qiao Jin, Yin Fang, Lauren He, Yifan Yang, Guangzhi Xiong, Zhizheng Wang, Nicholas Wan, Joey Chan, Donald C. Comeau, Robert Leaman, Charalampos S. Floudas, Aidong Zhang, Michael F. Chiang, Yifan Peng, Zhiyong Lu

llms nlp benchmarks safety

arXiv Machine Learning

Jun 2

LeAP: Learnable Adaptive Permutation for Feature Selection in Heterogeneous and Sparse Recommender Systems

arXiv:2606. 01111v1 Announce Type: new Abstract: Modern industrial recommender systems rely on thousands of heterogeneous features -- ranging from low-dimensional scalars (e.

By Yihong Huang, Chen Chu, Fei Chen, Yu Lin, Ruiduan Li, Zhihao Li

rag efficiency benchmarks

arXiv AI

Jun 2

The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

arXiv:2605. 09883v2 Announce Type: replace-cross Abstract: As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores genuinely reflect robust visual understanding?

By Xia Hu, Zhenrui Yue, Brian Potetz, Howard Zhou, Leonidas Guibas, Chun-Ta Lu, Zhicheng Wang

llms multimodal benchmarks

arXiv Machine Learning

Jun 2

ContinuousBench: Can Differentially Private Synthetic Text Improve Capabilities?

arXiv:2606. 01849v1 Announce Type: new Abstract: Differentially private (DP) text synthesis promises to unlock sensitive corpora for model training, but it remains unclear whether DP synthetic data transmits genuinely new knowledge and capabilities present only in those corpora.

By Peihan Liu, Lucas Rosenblatt, Weiwei Kong, Natalia Ponomareva, Gautam Kamath, Rachel Cummings, Roxana Geambasu, Yu Gan, Lillian Tsai, Alex Bie

benchmarks

arXiv Machine Learning

Jun 2

RADE: Random Add-Drop Edge as a Regularizer

arXiv:2606. 00757v1 Announce Type: new Abstract: Graph Neural Networks (GNNs) suffer from overfitting and over-squashing of long-range information.

By Danial Saber, Amirali Salehi-Abari

benchmarks safety

arXiv AI

Jun 2

SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes

arXiv:2606. 01912v1 Announce Type: new Abstract: Smart homes are evolving toward complex state-dependent living environments, requiring Large Language Models (LLMs) to reason over user intent, preferences, and multi-device interactions.

By Kuan Li, Shuo Zhang, Huacan Wang, Fangzhou Yu, Zecheng Sheng, Yi Gu, Weipeng Ming, Lei Xue, Chen Liu, Sen Hu, Ronghao Chen, Siyue Lin, Yuqing Hou, Xiaofeng Mou, Yi Xu

llms agents benchmarks

arXiv AI

Jun 2

Consistency evaluation of benchmarks used for causal discovery

arXiv:2606. 01789v1 Announce Type: new Abstract: In graphical causal model, causal discovery aims to construct a causal graph based on numerical data and domain knowledge in plain text.

By Yuzhe Zhang, Chihui Chen, Lina Yao, Chen Wang

llms benchmarks

arXiv Machine Learning

Jun 2

Accelerating physics-informed neural networks for full waveform inversion using a hybrid quantum-classical finite-basis architecture

arXiv:2606. 01110v1 Announce Type: cross Abstract: Full waveform inversion (FWI) reconstructs heterogeneous material properties from receiver data but remains computationally demanding.

By Hoang Anh Nguyen, Divakar Vashisth, Ali Tura

benchmarks

arXiv Machine Learning

Jun 2

Quantifying the Energy Floor: Direct Measurement and Replay Buffer Bias in SAC-Based HVAC Control on sbsim

arXiv:2606. 01665v1 Announce Type: new Abstract: We quantify the energy floor -- the minimum achievable cost given action space constraints -- for Soft Actor-Critic (SAC) HVAC control on the sbsim calibrated building simulator.

By Bo Li, Chen Zhang

benchmarks safety

arXiv AI

Jun 2

Multi-Agent Conformal Prediction with Personalized Statistical Validity

arXiv:2606. 00717v1 Announce Type: cross Abstract: Uncertainty quantification is essential in high-stakes machine learning tasks.

By Martin V. Vejling, Christophe A. N. Biscio, Adrien Mazoyer, Petar Popovski, Shashi Raj Pandey

agents benchmarks safety

arXiv Machine Learning

Jun 2

Reconsidering Positional Supervision in Masked Diffusion Language Model Training

arXiv:2601. 22947v2 Announce Type: replace-cross Abstract: Masked diffusion language models (MDLMs) generate text by unmasking tokens in parallel and have recently emerged as alternatives to autoregressive language models.

By Mengyu Ye, Keito Kudo, Ryosuke Takahashi, Jun Suzuki

llms diffusion fine-tuning benchmarks safety

arXiv AI

Jun 2

LALE: Lightweight-Transformer Architecture for Land-Cover Estimation

arXiv:2606. 02092v1 Announce Type: cross Abstract: Semantic segmentation of remote sensing imagery requires models that capture both global context and local detail under tight computational budgets.

By \"Umit Mert \c{C}a\u{g}lar, Alptekin Temizel

llms computer-vision efficiency benchmarks

arXiv AI

Jun 2

MindGames Arena Generalization Track: In2AI Solution with Delayed Per-Step Reward Attribution

arXiv:2606. 00017v1 Announce Type: new Abstract: Training language model agents for multi-agent strategic interaction presents a core difficulty: the quality of any action may depend on future events that never materialize, on moves that violate game rules, or on decisions made by other players.

By Aliaksei Korshuk, Alexander Buyantuev, Ilya Makarov

llms agents reinforcement-learning benchmarks

arXiv Machine Learning

Jun 2

GLENS: Global Search via Learning from Solver Iterates with Diffusion Models

arXiv:2606. 00366v1 Announce Type: new Abstract: We consider the problem of generating a large collection of initial guesses for local minima of multimodal non-convex continuous optimization problems.

By Anjian Li, Bartolomeo Stellato, Ryne Beeson

diffusion robotics multimodal benchmarks

arXiv AI

Jun 2

Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue

arXiv:2606. 01223v1 Announce Type: cross Abstract: Despite substantial progress in long-context modeling, existing benchmarks remain confined to factual memory for explicit recall, failing to measure the reflective memory required to synthesize fragmented, multimodal cues into high-level interpretations.

By Jingjie Lin, Bingbing Wang, Zihan Wang, Zhengda Jin, Weiming Qiao, Jing Li, Ruifeng Xu

multimodal benchmarks safety

arXiv Machine Learning

Jun 2

Quality Audio Prototyping: a prototype system for unified sound retrieval and procedural generation

arXiv:2606. 00629v1 Announce Type: cross Abstract: Sound design workflows frequently oscillate between time-consuming library searches and the complexity of procedural synthesis, with practitioners typically relying on disconnected tools to address each challenge separately.

By Nelly Garcia, Aditya Bhattacharjee, Gabryel Mason-Williams, Israel Mason-Williams, Emmanouil Benetos, Joshua Reiss

benchmarks

arXiv Machine Learning

Jun 2

A Biconvex Formulation for Stable Transport of Mixture Models with a Unique Solution

arXiv:2606. 02515v1 Announce Type: new Abstract: Optimal transport (OT) provides a principled framework for mapping between probability distributions.

By Yeganeh Marghi, Kelly Jin, Uygar S\"umb\"ul

benchmarks

Get the daily briefing