Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,147 stories · RSS feed

arXiv AI

Oct 7

OpenSplatGraph: From Dense Semantic Maps to Structured Scene Graphs for Open-Vocabulary Robot Perception

arXiv:2610.07569v1 Announce Type: cross Abstract: Dense 3D mapping with semantic understanding is essential for robotic perception in complex environments. Recent 3D Gaussian Splatting-based mapping...

By Binh Long Nguyen, Kien Nguyen, Clinton Fookes, Peyman Moghadam

robotics benchmarks

arXiv AI

Oct 7

Disentangling Dual Image References in Frequency Aware Diffusion Models for Personalized Generation

arXiv:2610.07684v1 Announce Type: cross Abstract: Personalized image generation aims to synthesize text-driven images conditioned on reference images, while mainly casting the generation as image cus...

By Haipeng Liu, Yang Wang, Meng Wang

diffusion benchmarks

arXiv AI

Oct 7

ServeLearnBench: How Well Can Agents Self-Improve from Serving Experience?

arXiv:2610.07792v1 Announce Type: cross Abstract: Large language model agents are increasingly deployed to perform complex tasks in real-world environments. However, the knowledge required for correc...

By Haizhong Zheng, Yizhuo Di, Ranajoy Sadhukhan, Shuowei Jin, Beidi Chen

llms rag agents benchmarks

arXiv AI

Oct 7

Dynamic Alignment and Calibration for Multimodal Learning

arXiv:2610.07928v1 Announce Type: cross Abstract: Dynamic multimodal learning aims to learn robust representations by adaptively modeling information discrepancies across modalities. However, existin...

By Jinghao Xu, Zhenhua Guo, Xiaofeng Zhu, Xiaoshuang Shi

multimodal benchmarks safety

arXiv AI

Oct 7

Symphony for Text Generation: Benchmarking Clinical Note Generation

arXiv:2610.08161v1 Announce Type: cross Abstract: Ambient documentation systems are rapidly gaining adoption, yet their impact on clinical note quality remains poorly characterized. We introduce MedC...

By Daniel Varab, Victor Petr\'en Bach Hansen, Asbj{\o}rn W. Helge, Kevin Pelgrims, Mathias Baltzersen, Adrian Young-San Roessler, Vanessa Klungtvedt, Maximilian Brand, Lasse Krogsb{\o}ll, Henrik Cullen, Lars Maal{\o}e

llms fine-tuning benchmarks

arXiv AI

Oct 7

Visual Abstention in Unified Multimodal Models

arXiv:2610.07887v1 Announce Type: cross Abstract: Unified multimodal models (UMMs) integrate understanding and generation, yet their generative behavior is rarely governed by what they understand abo...

By Chufan Shi, Cheng Yang, Tiannuo Yang, Isadora White, Yiwei Chen, Taylor Berg-Kirkpatrick, Xuezhe Ma

llms multimodal benchmarks

arXiv AI

Oct 7

GeoPID: Decomposing and Steering Visual Information in Vision-Language Models

arXiv:2610.08401v1 Announce Type: cross Abstract: While recent vision-language models (VLMs) have shown outstanding performance across diverse applications, they tend to under-use visual information...

By Seulgi Kim, Zhixiong Zhang, Xinwei Zhang, Jie Ling, Ronn Shaw

llms multimodal benchmarks

arXiv AI

Oct 7

X-OPM: Explainable Automatic Digital On-Chip Power Modeling for Enhanced Robustness

arXiv:2610.08502v1 Announce Type: cross Abstract: Proactive power management systems reduce processor dynamic power through runtime power prediction and power-aware scheduling. Accurate, stable and l...

By Jingbo Jiang, Xizi Chen, Jian Peng, Wei Zhang

benchmarks safety

arXiv AI

Oct 7

RAG-PIBench: A Leakage-Aware Benchmark for Prompt-Injection Detection in Trustworthy RAG Systems

arXiv:2610.08571v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) systems are vulnerable to prompt-injection attacks embedded in retrieved content. We introduce RAG-PIBench, a be...

By Niveen O. Jaffal, Ahmet Yuksel, David Mohaisen

llms rag benchmarks

arXiv AI

Oct 7

HygieneRoboBench: Benchmarking Hygiene-Aware Planning for Household Robots

arXiv:2610.08642v1 Announce Type: cross Abstract: Contact with contaminated objects can spread hazards through a household robot's grippers, tools, and shared surfaces, while new contacts can make an...

By Yurun Chen, Josh Qixuan Sun, Jason Qin, Chengtai Li, Tianyi Wang, Mark Crowley, Wentao Zhu

llms robotics benchmarks

arXiv AI

Oct 7

Secure Speculative Decoding for Large Language Models

arXiv:2610.08678v1 Announce Type: cross Abstract: Speculative decoding accelerates inference for a large language model (LLM), referred to as the \emph{target model}, by first using a smaller model,...

By Yichi Zhang, Zhiqi Wang, Neil Gong, Yuchen Yang

llms benchmarks safety

arXiv AI

Oct 7

When Explanations Compete: Policy-Aware Selection Under Uncertainty

arXiv:2410.05479v2 Announce Type: replace Abstract: Uncertainty-aware explanation methods often produce several alternatives for the same prediction. Selecting among them requires a policy for balanc...

By Helena L\"ofstr\"om, Tuwe L\"ofstr\"om, Johan Hallberg Szabadvary

benchmarks

arXiv AI

Oct 7

SDFlow: Similarity-Driven Flow Matching for Time Series Generation

arXiv:2605.05736v3 Announce Type: replace Abstract: Vector quantization (VQ) with autoregressive (AR) token modeling is a widely adopted and highly competitive paradigm for time-series generation. Ho...

By Wei Li, Shibo Feng, Pengcheng Wu, Xingyu Gao, Min Wu, Peilin Zhao

diffusion efficiency benchmarks safety

arXiv AI

Oct 7

SpecFold: Folding Multi-Branch Redundancy for Faster Speculative Decoding in Diffusion Language Models

arXiv:2610.04875v2 Announce Type: replace Abstract: Diffusion large language models (DLLMs) generate text through iterative block denoising, and multi-branch speculative decoding accelerates this pro...

By Chung-En Ho, Weiyu Sun, Cheng-Jhih Shih, He Li, Yong Liu, Yingyan Celine Lin

llms diffusion efficiency benchmarks

arXiv AI

Oct 7

Hypergraph-Enhanced Dual Convolutional Network for Bundle Recommendation

arXiv:2312.11018v3 Announce Type: replace-cross Abstract: Bundle recommendation ranks sets of related items rather than isolated items. Its central challenge is to connect user preferences, item inte...

By Yang Li, Kangbo Liu, Yaoxin Wu, Zhaoxuan Wang, Erik Cambria

benchmarks

arXiv AI

Oct 7

Large Pretraining Datasets Don't Guarantee Robustness after Fine-Tuning in Image Classification

arXiv:2410.21582v4 Announce Type: replace-cross Abstract: Large-scale pretrained models are widely leveraged as foundations for learning new specialized tasks via fine-tuning, with the goal of mainta...

By Jaedong Hwang, Brian Cheung, Zhang-Wei Hong, Akhilan Boopathy, Pulkit Agrawal, Ila Fiete

computer-vision fine-tuning benchmarks

arXiv AI

Oct 7

Time-o1: Time-Series Forecasting Needs Transformed Label Alignment

arXiv:2505.17847v3 Announce Type: replace-cross Abstract: Training time-series forecasting models poses unique challenges in loss function design. Most existing approaches adopt temporal mean squared...

By Hao Wang, Licheng Pan, Zhichao Chen, Xu Chen, Qingyang Dai, Lei Wang, Haoxuan Li, Zhouchen Lin

benchmarks safety

arXiv AI

Oct 7

Real-Time Generation of Game Video Commentary with Multimodal LLMs: Pause-Aware Decoding Approaches

arXiv:2603.02655v2 Announce Type: replace-cross Abstract: Real-time video commentary generation provides textual descriptions of ongoing events in videos. It supports accessibility and engagement in...

By Anum Afzal, Yuki Saito, Hiroya Takamura, Katsuhito Sudoh, Shinnosuke Takamichi, Graham Neubig, Florian Matthes, Tatsuya Ishigaki

llms fine-tuning multimodal benchmarks

arXiv AI

Oct 7

Federated Mixture-of-Experts Alignment on Mobile Edge Networks under Data Heterogeneity

arXiv:2603.21276v2 Announce Type: replace-cross Abstract: The growing demand for on-device large language model (LLM) services on mobile edge devices has driven the adoption of Mixture-of-Experts (Mo...

By Zihan Fang, Qianru Wang, Haonan An, Zheng Lin, Yiqin Deng, Symeon Chatzinotas, Yuguang Fang

llms fine-tuning benchmarks safety

arXiv AI

Oct 7

Learning Visual Feature-Based World Models via Residual Latent Action

arXiv:2605.07079v2 Announce Type: replace-cross Abstract: World models predict future transitions from observations and actions. Existing works predominantly focus on image generation only. Visual fe...

By Xinyu Zhang, Zhengtong Xu, Yutian Tao, Yeping Wang, Yu She, Abdeslam Boularias

diffusion reinforcement-learning robotics benchmarks safety

Get the daily briefing