Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

22,245 stories · RSS feed

arXiv AI

Jun 10

Representation Curriculum: Stagewise Training for Robust Ranking and Allocation

arXiv:2606. 09891v1 Announce Type: cross Abstract: Ranking in digital marketplaces is a dynamic exposure-allocation mechanism: displayed items shape discovery trajectories and success events logged by the platform to update future allocation policies.

By Ehsan Ebrahimzadeh, Sina Baharlouei, Abraham Bagherjeiran

benchmarks

arXiv AI

Jun 10

When Attribution Patching Lies: Diagnosis and a Second-Order Correction

arXiv:2606. 09899v1 Announce Type: cross Abstract: A central goal of mechanistic interpretability is to identify which internal components causally drive a language model's behavior.

By Luyang Zhang, Jialu Wang

llms benchmarks safety

arXiv Machine Learning

Jun 10

On the Controllability-Fidelity Frontier in Diffusion Editing

arXiv:2606. 09901v1 Announce Type: cross Abstract: Diffusion-based generative models enable powerful image editing capabilities, but achieving precise control while maintaining fidelity and safety remains challenging.

By Yi Hu, Leying Yi, Emily Davis, Finn Carter

diffusion robotics benchmarks safety

arXiv AI

Jun 10

Sample Where You Struggle: Sharpening Base Model Reasoning via Entropy-Guided Power Sampling

arXiv:2606. 09926v1 Announce Type: cross Abstract: Sampling from the sequence-level power distribution $p^\alpha$ elicits RL-level reasoning from base language models without any parameter updates, but the standard Metropolis--Hastings (MH), a Markov Chain Monte Carlo (MCMC) sampler, is both expensive and slow-mixing.

By Hong Guo, Nianhui Guo, Christoph Meinel, Haojin Yang

llms reinforcement-learning benchmarks

arXiv AI

Jun 10

RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty

arXiv:2602. 12424v2 Announce Type: replace-cross Abstract: Benchmarks establish a standardized evaluation framework to systematically assess the performance of large language models (LLMs), facilitating objective comparisons and driving advancements in the field.

By Ziqian Zhang, Xingjian Hu, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun

llms benchmarks

arXiv AI

Jun 10

HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning

arXiv:2606. 10507v1 Announce Type: new Abstract: While Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents across a wide range of tasks, their performance often degrades in multi-turn long-horizon agentic tasks.

By Juncheng Diao, Zhicong Lu, Peiguang Li, Yongwei Zhou, Changyuan Tian, Qingbin Li, Rongxiang Weng, Jingang Wang, Xunliang Cai

llms agents nlp reinforcement-learning benchmarks

arXiv AI

Jun 10

MoE Enhanced Federated Learning for Spatiotemporal Prediction

arXiv:2606. 10499v1 Announce Type: cross Abstract: Traffic prediction is fundamental to intelligent transportation systems and urban computing, yet many cities continue to suffer from traffic data scarcity due to limited sensor deployment and uneven urban development.

By Zhehao Dai, Xiao Han, Zhaolin Deng, Zijian Zhang, Xiangyu Zhao, Guojiang Shen, Xiangjie Kong

benchmarks safety

arXiv Machine Learning

Jun 10

Domain Adapted Large Language Models for Additive Manufacturing

arXiv:2603. 22017v2 Announce Type: replace Abstract: This work presents a collection of multi-modal domain adapted large language models built upon the instruction tuned variants of open weight models (Gemma 3, Qwen 3, Gemma 4) using a relatively small dataset of around 50 million tokens.

By Peter Pak, Amir Barati Farimani

llms fine-tuning multimodal benchmarks

arXiv AI

Jun 10

ActiveMem: Distributed Active Memory for Long-Horizon LLM Reasoning

arXiv:2606. 10532v1 Announce Type: new Abstract: Memory is essential for enabling large language model (LLM) agents to handle long-horizon reasoning tasks.

By Yunhan Jiang, Wenbin Duan, Shasha Guo, Liang Pang, Xiaoqian Sun, Huawei Shen

llms agents efficiency benchmarks

arXiv AI

Jun 10

Atomic Intent Reasoning: Bringing LLM Semantics to Industrial Cross-Domain Recommendations

arXiv:2606. 10357v1 Announce Type: cross Abstract: Cross-domain recommendation is a core problem in content-to-e-commerce platforms.

By Zhuohang Jiang, Yuxin Chen, Shijie Wang, Haohao Qu, Zhou Jindong, Wenqi Fan, Li Qing, Dongxu Liang, Jun Wang

llms efficiency benchmarks

arXiv Machine Learning

Jun 10

Overcoming Rank Collapse in Feedback Alignment

arXiv:2606. 11123v1 Announce Type: new Abstract: Backpropagation (BP) is widely viewed as biologically implausible, in part because it requires feedback weights to be the transpose of forward weights for error propagation.

By Gauthier Boeshertz, Razvan Pascanu, Claudia Clopath

benchmarks safety

arXiv Machine Learning

Jun 10

Closing the Modality Gap in Zero-Shot HAR: Contrastive Training and Separability-Optimized Prototypes on IMU Data

arXiv:2606. 10789v1 Announce Type: new Abstract: Zero-shot learning (ZSL) for inertial measurement unit (IMU)-based human activity recognition (HAR) faces a central challenge: bridging the gap between sensor embeddings and semantic class representations.

By Anik Ghosh

llms rag benchmarks safety

arXiv Machine Learning

Jun 10

When Do Autoregressive Sequence Models Forecast Physical Wavefields? A Controlled Study on Synthetic Seismograms

arXiv:2606. 10868v1 Announce Type: new Abstract: Long-horizon autoregressive forecasting of oscillatory physical signals, such as seismograms, gravitational-wave strain, and similar wavefields is limited by error accumulation: as a causal model is fed its own outputs over hundreds of steps, small per-step errors compound into phase drift that pointwise metrics fail to detect.

By Waleed Esmail, Stuart Russell, Jana Klinge, Alexander Kappes, Christine Thomas

rag benchmarks

arXiv AI

Jun 10

TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning

arXiv:2606. 11119v1 Announce Type: cross Abstract: Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.

By Heming Zou, Qi Wang, Yun Qu, Yuhang Jiang, Lizhou Cai, Yixiu Mao, Ru Peng, Xin Xu, Weijie Liu, Kai Yang, Saiyong Yang, Xiangyang Ji

llms agents reinforcement-learning benchmarks

arXiv AI

Jun 10

Density Ridge Selective Prediction for LLM and VLM Hallucination Detection under Calibration Label Scarcity

arXiv:2606. 10198v1 Announce Type: cross Abstract: Hallucination detection in large language and vision-language models is increasingly framed as selective prediction, where a detector assigns a confidence score and abstains when confidence is low.

By Nina I. Shamsi

llms multimodal benchmarks safety

arXiv AI

Jun 10

SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs

arXiv:2606. 09886v1 Announce Type: cross Abstract: Sparse Mixture-of-Experts (MoE) large language models achieve strong quality with low per-token compute, yet their deployment is often limited by the memory wall: the full expert pool must remain resident to support token-dependent routing.

By Yuhao Zhang

llms efficiency benchmarks

arXiv Machine Learning

Jun 10

When Design Rules Break: Benchmark Composition Determines Whether Label Informativeness Predicts GNN Aggregator Choice

arXiv:2606. 10249v1 Announce Type: new Abstract: We examine whether graph neural network (GNN) design rules generalize across benchmark families by studying aggregator selection (sum, mean, max) on 24 node-classification datasets spanning citation, heterophilic, LINKX Facebook-100, co-purchase, and co-authorship graphs.

By Neha Sharma, Ritesh Sharma

benchmarks

arXiv AI

Jun 10

One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QA

arXiv:2606. 10572v1 Announce Type: new Abstract: External memory effectively grounds large language models (LLMs) and vision-language models (VLMs)-based question answering (QA) in relevant multimodal evidence.

By Zhi Zheng, Ziqiao Meng, Hao Luan, Wei Liu, Wee Sun Lee

llms rag nlp efficiency multimodal benchmarks

arXiv Machine Learning

Jun 10

One Step Closer to Ground Truth: A Multi-Scale Residual-Aware Representation Learning Pipeline for Predicting Time Series Data

arXiv:2606. 10678v1 Announce Type: new Abstract: Transformer-based models have emerged as leading paradigms in time-series forecasting in recent years, employing self-attention mechanisms to capture long-range dependencies.

By Amrijit Biswas, Mustafa Kamal, Robin Krambroeckers, M. M. Lutfe Elahi, Sifat Momen, Nabeel Mohammed, Shafin Rahman

llms benchmarks safety

arXiv Machine Learning

Jun 10

Interpretable deep convolutional model for nonlinear multivariate time series in complex systems

arXiv:2501. 04339v2 Announce Type: replace-cross Abstract: We introduce the Deep Convolutional Interpreter for Time Series (DCIts), a deep-learning architecture for nonlinear multivariate time series that provides sample-specific, locally interpretable descriptions of the underlying interaction structure.

By Domjan Baric, Davor Horvatic

benchmarks safety

Get the daily briefing