Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,077 stories · RSS feed

arXiv AI

Jul 20

Beyond a Joke: Multi-Angle Reasoning for Detecting and Explaining Harmful Humor in Memes

arXiv:2607. 15442v1 Announce Type: new Abstract: Internet memes intertwine visual cues, textual content, and cultural context, making them particularly challenging to interpret in scenarios where humor, sarcasm, and harmful intent coexist.

By Shanhong Liu, Pai Chet Ng, De Wen Soh, Malika Meghjani, Konstantinos N. Plataniotis

llms multimodal benchmarks safety

arXiv AI

Jul 20

Neuro-Symbolic AI for LEED compliance: Document-Centric Benchmarking, Deterministic Numeric Checking, and When Multimodal Hurts

arXiv:2607. 15647v1 Announce Type: new Abstract: LEED v4.

By Aritro De (The University of Texas at Austin), Juliana Felkner (The University of Texas at Austin)

llms multimodal benchmarks

arXiv AI

Jul 20

When Do Multi-Agent Systems Help? An Information Bottleneck Perspective

arXiv:2607. 16133v1 Announce Type: cross Abstract: LLM powered multi-agent systems (MAS) have emerged as a promising paradigm for complex tasks.

By Wendi Yu, Lianhao Zhou, Xiangjue Dong, Sai Sudarshan Barath, Declan Staunton, Byung-Jun Yoon, Xiaoning Qian, James Caverlee, Shuiwang Ji

llms agents benchmarks

arXiv Machine Learning

Jul 20

QUADS: Stabilizing NVFP4 Reinforcement Learning for MoE via QUantization-error Alignment across Dual Sides

arXiv:2607. 15810v1 Announce Type: new Abstract: Rollout generation is a major bottleneck in Reinforcement Learning (RL) for Mixture-of-Experts (MoE) Large Language Models, motivating low-precision rollout acceleration such as FP8.

By Zhengyang Zhuge, Hao Yu, Xin Wang, Zheng Li, Yizhong Cao, Dayiheng Liu, Jianwei Zhang

llms reinforcement-learning efficiency benchmarks safety

arXiv Machine Learning

Jul 20

ContinuityBench: A Benchmark and Systems Study of Stateful Failover in Multi-Provider LLM Routing

arXiv:2607. 15899v1 Announce Type: new Abstract: In production large language model (LLM) deployments, high API availability guarantees do not equate to conversational continuity.

By Vishal Pandey, Gopal Singh

llms benchmarks

arXiv Machine Learning

Jul 20

Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems

arXiv:2607. 16177v1 Announce Type: new Abstract: Reinforcement learning (RL) has recently emerged as a promising feedback control strategy for nonlinear and complex dynamical systems.

By Matteo Tomasetto, Nicol\`o Botteghi, Gabriele Bruni, Andrea Manzoni

agents reinforcement-learning benchmarks

arXiv Machine Learning

Jul 20

DyneTrion: A Spatio-temporally Coherent Generative Emulator for Protein Dynamics Across Timescales

arXiv:2607. 15309v1 Announce Type: cross Abstract: Proteins function through coordinated motion across multiple spatial and temporal scales, underpinning processes such as ligand binding, allostery, and catalysis.

By Kaihui Cheng, Zhiqiang Cai, Peng Tu, Yisong Yao, Limei Han, Libo Wu, Siyu Zhu, Tzuhsiung Yang, Yuan Qi

benchmarks

arXiv Machine Learning

Jul 20

VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

arXiv:2607. 15498v1 Announce Type: cross Abstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference.

By Shahrzad Esmat, Dhawal Shah, Ali Jannesari

llms efficiency benchmarks

arXiv AI

Jul 20

SLAC: Safe and Efficient Real-Robot Reinforcement Learning via Unsupervised Simulation Pre-Training

arXiv:2506. 04147v5 Announce Type: replace-cross Abstract: Building capable household and industrial robots requires mastering the control of versatile, high-degree-of-freedom (DoF) systems such as mobile manipulators.

By Jiaheng Hu, Peter Stone, Roberto Mart\'in-Mart\'in

reinforcement-learning robotics benchmarks safety

arXiv AI

Jul 20

Jailbreak Foundry: From Papers to Runnable Attacks for Reproducible Benchmarking

arXiv:2602. 24009v4 Announce Type: replace-cross Abstract: Jailbreak techniques for large language models (LLMs) evolve faster than benchmarks, making robustness estimates stale and difficult to compare across papers due to drift in datasets, harnesses, and judging protocols.

By Zhicheng Fang, Jingjie Zheng, Chenxu Fu, Wei Xu

llms agents multimodal benchmarks safety

arXiv AI

Jul 20

DrawingVQA: A Real-World Benchmark for Multi-Depth Visual-Textual Reasoning on Construction Drawings

arXiv:2607. 15418v1 Announce Type: new Abstract: We introduce DrawingVQA, the first benchmark designed to evaluate multimodal large language models (MLLMs) on real-world construction drawings -- a core media in architecture, civil, and many other engineering practices.

By Yoonhwa Jung, Junryu Fu, Mani Golparvar-Fard

llms multimodal benchmarks

arXiv Machine Learning

Jul 20

LLM4EHR: Aligning Clinical Time Series with Medical Event Sequences via Large Language Models

arXiv:2607. 15447v1 Announce Type: new Abstract: Recent research in clinical machine learning, focusing on outcome predictions in intensive care unit (ICU), has shifted from bespoke supervised models to foundation models, utilising modern representation learning methods.

By Jingteng Li, Alexander Capstick, Louise Rigny, Iona Biggart, Neil J Sebire, Payam Barnaghi

llms rag benchmarks

arXiv Machine Learning

Jul 20

(MPO)$^2$: Multivariate Polynomial Optimization based on Matrix Product Operators

arXiv:2607. 15916v1 Announce Type: new Abstract: Central to machine learning and signal processing is the ability to perform universal function approximation and learn complex input-output relationships from limited numbers of observations.

By Niccol\`o Ciolli, Anders Vestergaard N{\o}rskov, Michael Kastoryano, Petr Taborsky, Morten M{\o}rup

rag benchmarks

arXiv AI

Jul 20

MGDT: MLLM-Guided Diffusion Transformer with Relation-Adaptive Mixture-of-Experts for Multimodal Knowledge Graph Completion

arXiv:2607. 15592v1 Announce Type: new Abstract: Multimodal Knowledge Graph Completion (MKGC) requires inferring missing entities from structural, textual, and visual cues.

By Xu Hou, Meiyu Liang, Wei Huang, Yawen Li, Zhe Xue, Wu Liu, Guanhua Ye, Lei Shi, Kangkang Lu

llms diffusion multimodal benchmarks safety

arXiv Machine Learning

Jul 20

Improving Backward Conformal Prediction via Non-Conformity Score Transformation

arXiv:2602. 01733v3 Announce Type: replace-cross Abstract: Conformal Prediction (CP) provides a statistical framework for uncertainty quantification that constructs prediction sets with coverage guarantees.

By Junxian Liu, Hao Zeng, Hongxin Wei

benchmarks

arXiv AI

Jul 20

Loop the Loopies!

arXiv:2607. 16051v1 Announce Type: cross Abstract: We present Loopie, the most powerful looped Transformer to date.

By Zitian Gao, Yilong Chen, Yihao Xiao, Xinyu Yang, Ran Tao, Joey Zhou, Bryan Dai

llms benchmarks

arXiv Machine Learning

Jul 20

ASK-NN: An Asymmetric Nearest-Neighbor Test that detects Distribution Drifts in Natural Language

arXiv:2607. 15607v1 Announce Type: new Abstract: Hallucinations and artificial text in LLM-generated outputs often appear as distributional deviations between prompt and response hidden-state distributions.

By Sergey Zakharov, Rodion Oblovatny, Alexey Zaytsev

llms benchmarks safety

arXiv AI

Jul 20

Scaling Time Series Classification via XAI-Driven Data Reduction

arXiv:2607. 15774v1 Announce Type: cross Abstract: Explainable AI (XAI) for time series has seen significant algorithmic growth, but its utility in providing measurable performance gains for downstream tasks remains under-explored.

By Davide Italo Serramazza, Thach Le Nguyen, Georgiana Ifrim

llms benchmarks safety

arXiv AI

Jul 20

LVSum: A Benchmark for Timestamp-Aware Long Video Summarization

arXiv:2604. 10024v2 Announce Type: replace-cross Abstract: Long video summarization presents significant challenges for multimodal large language models (MLLMs), particularly in maintaining temporal fidelity over extended durations and producing summaries that are both semantically and temporally grounded.

By Alkesh Patel, Melis Ozyildirim, Ying-Chang Cheng, Ganesh Nagarajan

llms nlp multimodal benchmarks safety

arXiv AI

Jul 20

Orbis 2: A Hierarchical World Model for Driving

arXiv:2607. 15898v1 Announce Type: cross Abstract: Current world models operate at a single level of abstraction, with most prioritizing perceptual fidelity while lacking the spatial reasoning and semantic understanding required for real-world downstream tasks.

By Sudhanshu Mittal, Arian Mousakhan, Silvio Galesso, Karim Farid, Jonannes Dienert, Rajat Sahay, Thomas Brox

diffusion benchmarks

Get the daily briefing