Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

21,561 stories · RSS feed

arXiv AI

Jun 16

AdaSTORM: Scaling LLM Reasoning on Dynamic Graphs via Adaptive Spatio-Temporal Multi-Agent Collaboration

arXiv:2606. 16328v1 Announce Type: new Abstract: Large Language Models (LLMs) demonstrate remarkable potential in dynamic graph reasoning, but suffer from a scaling bottleneck: current models can only handle graphs with tens of nodes, constrained by exponential reasoning overhead and finite context windows.

By Bing Hao, Ruijie Wang, Haodong Qian, Yunlong Chu, Yuhang Liu, Yumeng Lin, Minglai Shao, Jianxin Li

llms agents benchmarks

arXiv Machine Learning

Jun 16

Evaluating LLM Personalization via Semantic Constraint Verification

arXiv:2606. 16368v1 Announce Type: cross Abstract: Current evaluation paradigms for Large Language Model (LLM) personalization rely heavily on brittle surface-matching metrics or computationally expensive LLM-as-a-judge protocols, both of which lack interpretability.

By Xuran Li, Guanqin Zhang, Imran Razzak, Hakim Hacid, Eleanna Kafeza, Hao Xue, Flora D. Salim

llms benchmarks safety

arXiv AI

Jun 16

K-Prism: A Knowledge-Guided and Prompt Integrated Universal Medical Image Segmentation Model

arXiv:2509. 25594v2 Announce Type: replace-cross Abstract: Medical image segmentation is fundamental to clinical decision-making, yet existing models remain fragmented.

By Bangwei Guo, Yunhe Gao, Meng Ye, Difei Gu, Yang Zhou, Leon Axel, Dimitris Metaxas

computer-vision benchmarks

arXiv AI

Jun 16

LabOSBench: Benchmarking Computer Use Agents for Scientific Instrument Control

arXiv:2606. 16802v1 Announce Type: new Abstract: Current computer-use benchmarks primarily focus on software operation tasks in virtualized systems, whereas scientific instrumentation scenarios require coordinated control over complex interfaces, and feedback-driven parameter adjustment.

By Anqi Zou, Han Deng, Chengyu Zhang, Junquan Hu, Yu Wang, Yuxiang Xing, Aokai Zhang, Hanling Zhang, Zhaoyang Liu, Ben Fei, Zhihui Wang, Wanli Ouyang

llms agents multimodal benchmarks safety

arXiv AI

Jun 16

Evaluating the Robustness of Proof Autoformalization in Lean 4

arXiv:2606. 14867v1 Announce Type: cross Abstract: Proof autoformalization aims to translate a mathematical informal proof written in natural language into a formal proof in a formal language such as Lean~4.

By Zhengtao Gui, Sheng Yang, Zhouxing Shi

llms benchmarks

arXiv AI

Jun 16

Theorem-Grounded Execution Ontologies for Interpretable Machine Reasoning

arXiv:2606. 16010v1 Announce Type: cross Abstract: Large language models have achieved impressive performance on reasoning tasks spanning mathematics, science, programming, and commonsense inference.

By Raghu Anantharangachar

llms benchmarks

arXiv AI

Jun 16

Haiku to Opus in Just 10 bits: LLMs Unlock Large Compression Gains

arXiv:2604. 02343v2 Announce Type: replace-cross Abstract: We study the compression of LLM-generated text across lossless and lossy regimes, characterizing a compression-compute frontier where more compression is possible at the cost of more compute.

By Roy Rinberg, Annabelle Michael Carrell, Simon Henniger, Nicholas Carlini, Keri Warr

llms fine-tuning benchmarks

arXiv AI

Jun 16

Think-at-Hard: Selective Latent Iterations to Improve Reasoning Language Models

arXiv:2511. 08577v3 Announce Type: replace-cross Abstract: Improving the reasoning abilities of Large Language Models (LLMs), especially under parameter constraints, is crucial for real-world applications.

By Tianyu Fu, Yichen You, Zekai Chen, Guohao Dai, Huazhong Yang, Yu Wang

llms fine-tuning benchmarks

arXiv AI

Jun 16

Bridging the Gap: Enabling Natural Language Queries for NoSQL Databases through Text-to-NoSQL Translation

arXiv:2502. 11201v3 Announce Type: replace-cross Abstract: NoSQL databases are core data infrastructure, yet natural-language access to them remains underdeveloped: correct query generation must recover how a non-relational data model represents entities, nested paths, arrays, missing fields, and dynamic keys.

By Jinwei Lu, Jiawei Lu, Chen Zhang, Zhiqian Qin, Haodi Zhang, Yuanfeng Song, Raymond Chi-Wing Wong

llms benchmarks data-engineering

arXiv AI

Jun 16

Safe Exploration via Policy Priors

arXiv:2601. 19612v3 Announce Type: replace-cross Abstract: Safe exploration is a key requirement for reinforcement learning (RL) agents to learn and adapt online, beyond controlled (e.

By Manuel Wendl, Yarden As, Manish Prajapat, Anton Pollak, Stelian Coros, Andreas Krause

agents reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 16

PhysGuard: Fisher-Guided Gradient Projection for Sim-to-Real Neural PDE Surrogates

arXiv:2606. 16602v1 Announce Type: new Abstract: Neural operator models trained on simulation data often lose accuracy when applied to experimental measurements due to the sim-to-real gap.

By Changjian Zhou, Junfeng Fang, Negin Yousefpour, Peng Wu, Bin Yan, Guillermo A Narsilio

robotics fine-tuning benchmarks

arXiv Machine Learning

Jun 16

Think Less, Act Early: Reinforced Latent Reasoning with Early Exit in Vision-Language-Action Models

arXiv:2606. 15099v1 Announce Type: cross Abstract: Existing Vision-Language-Action (VLA) models predominantly rely on explicit Chain-of-Thought (CoT) reasoning to bridge perception and action.

By Dianqiao Lei, Lianlei Shan

reinforcement-learning robotics multimodal benchmarks safety

arXiv Machine Learning

Jun 16

Distilling latent electrostatics from foundation machine learning interatomic potentials

arXiv:2606. 15001v1 Announce Type: cross Abstract: Foundation machine learning interatomic potentials (MLIPs) have enabled atomistic simulations across broad regions of chemical and materials space, but many remain computationally expensive and lack explicit electrostatics, limiting their use for systems governed by long-range interactions and electrical response.

By Xiaoyu Wang, Bingqing Cheng

fine-tuning efficiency benchmarks

arXiv AI

Jun 16

RL-Index: Reinforcement Learning for Retrieval Index Reasoning

arXiv:2606. 16316v1 Announce Type: cross Abstract: Retrieving external knowledge is essential for solving real-world tasks, yet it remains challenging when the relationship between a query and its relevant knowledge involves implicit and complex reasoning beyond surface-level semantic or lexical matching (e.

By Yongjia Lei, Nedim Lipka, Zhisheng Qi, Utkarsh Sahu, Koustava Goswami, Franck Dernoncourt, Ryan A. Rossi, Yu Wang

llms agents reinforcement-learning efficiency benchmarks

arXiv AI

Jun 16

Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?

arXiv:2511. 20710v2 Announce Type: replace-cross Abstract: In the age of agentic AI, the growing deployment of multi-modal models (MMs) has introduced new attack vectors that can leak sensitive training data in MMs, causing privacy leakage.

By David Amebley, Sayanton Dibbo

llms agents multimodal benchmarks safety

arXiv Machine Learning

Jun 16

Localized Kernel Projection Outlyingness: A Two-Stage Approach for Multi-Modal Outlier Detection

arXiv:2510. 24043v4 Announce Type: replace Abstract: This paper presents Two-Stage LKPLO, a novel multi-stage outlier detection framework that overcomes the coexisting limitations of conventional projection-based methods: their reliance on a fixed statistical metric and their assumption of a single data structure.

By Akira Tamamori

multimodal benchmarks

arXiv Machine Learning

Jun 16

David vs. Goliath in Next Activity Prediction: Argmax vs. LSTM, Transformer, and LLM

arXiv:2606. 15868v1 Announce Type: new Abstract: Next activity prediction (NAP) is a cornerstone of predictive process monitoring (PPM), enabling organizations to move from retrospective analysis to proactive process steering.

By Hans Weytjens, Ingo Weber

llms benchmarks

arXiv Machine Learning

Jun 16

EIBench: A Simulator-Based Benchmark and Turn-Credit RL for Emotion Management

arXiv:2606. 15532v1 Announce Type: cross Abstract: Emotional intelligence (EI) in Large Language Models (LLMs) is often evaluated through static understanding tasks or single-response dialogue generation.

By Rongzhi Zhu, Xiang Huang, Yuchuan Wu, Rui Wang, Zequn Sun, Tao Ren, Weiyao Luo, Bingxue Qiu, Jieping Ye, Yongbin Li, Wei Hu

llms reinforcement-learning benchmarks

arXiv AI

Jun 16

No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networks

arXiv:2405. 02369v2 Announce Type: replace-cross Abstract: In the past decade, many successful networks are on novel architectures, which almost exclusively use the same type of neurons.

By Feng-Lei Fan, Meng Wang, Hang-Cheng Dong, Jianwei Ma, Tieyong Zeng

benchmarks safety

arXiv AI

Jun 16

Efficient Flow Matching using Latent Variables

arXiv:2505. 04486v4 Announce Type: replace-cross Abstract: Flow matching models have shown great potential in image generation tasks among probabilistic generative models.

By Anirban Samaddar, Yixuan Sun, Viktor Nilsson, Sandeep Madireddy

diffusion multimodal benchmarks safety

Get the daily briefing