Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv AI

Jul 17

BrainPilot: Automating Brain Discovery with Agentic Research

arXiv:2607. 15079v1 Announce Type: new Abstract: Understanding the brain increasingly depends on integrating evidence across scales, modalities, and disciplines.

By Haoxuan Li, Tianci Gao, Jianhe Li, Yang Fan, Runze Shi, Weiran Wang, Tianxiang Zhao, Zezhao Wu, Xiaoyang Jiang, Qihui Zhang, Jia Li, Xiao Xiao, Kai Du, Xiaoxuan Jia, Chao Xie, Lu Mi

rag agents benchmarks

arXiv AI

Jul 17

DialogueVPR: Towards Conversational Visual Place Recognition

arXiv:2607. 14115v1 Announce Type: new Abstract: Inspired by how humans communicate spatial information, language-guided geo-localization has gained significant traction for its intuitive and practical value.

By Yukun Song, Changwei Wang, Xingtian Pei, Shibiao Xu, Wenhao Xu, Shunpeng Chen, Yu Zhang, Ke Zhang, Rongtao Xu, Xuxiang Feng, Pengyang Wang

fine-tuning multimodal benchmarks

arXiv Machine Learning

Jul 17

DiMaS: Distribution Matching for Steering Vision-Language-Action Models

arXiv:2607. 14280v1 Announce Type: cross Abstract: Flow-matching-based vision-language-action (VLA) models have emerged as powerful policies for robotic manipulation, yet a critical capability remains underexplored: fine-grained behavioral control, the ability to govern how a robot performs a task by intervening on its internal representations.

By Pegah Khayatan, Sara Meziane, Jayneel Parekh, Matthieu Cord

llms robotics multimodal benchmarks safety

arXiv Machine Learning

Jul 17

Primal-dual algorithm for contextual stochastic combinatorial optimization

arXiv:2505. 04757v2 Announce Type: replace Abstract: This paper introduces a novel approach to contextual stochastic optimization, integrating operations research and machine learning to address decision-making under uncertainty.

By Louis Bouvier, Thibault Prunet, Vincent Lecl\`ere, Axel Parmentier

benchmarks

arXiv AI

Jul 17

StructureClaw: Traceable LLM Agents and an Executable Benchmark for Structural Engineering Workflows

arXiv:2607. 14896v1 Announce Type: cross Abstract: Addressing a structural-engineering request requires more than a single answer; it requires a chain of interdependent artifacts: interpreted requirements, a computable model, validation records, solver outputs, code-check records, and a final report.

By Sizhong Qin, Yi Gu, Yao Jiang, Ao Cai, Changjian Zhou, Shaoxuan Shuai, Jiachang Wang, Tianhao Shen, Yueqiang Li, Xinhao Li, Li Zeng, Yueshi Chen, Dachen Gao, Genrong Xu, Wenjie Liao, Xinzheng Lu

llms agents nlp multimodal benchmarks

arXiv Machine Learning

Jul 17

LATTICE: Graph Self-Supervised Learning for Multimodal Spatial Omics Integration

arXiv:2607. 14410v1 Announce Type: new Abstract: Spatially resolved omics studies increasingly combine transcriptomic and epigenomic assays, yet downstream analysis is often still performed using single-modality pipelines.

By Jagan Mohan Reddy Dwarampudi, Veena Kochat, Suresh Satpati, Kunal Rai, Tania Banerjee

rag multimodal benchmarks safety

arXiv AI

Jul 17

SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agents

arXiv:2607. 14543v1 Announce Type: cross Abstract: Vision-language models (VLMs) are increasingly used as the reasoning backbone of embodied agents, enabling robots to interpret visual scenes, follow language instructions, and plan multi-step actions.

By Huaigang Yang, Ya Li, Min Ren, Bo Dai, Zhenliang Zhang, Zhaofeng He

llms agents robotics multimodal benchmarks safety

arXiv AI

Jul 17

Towards Hierarchical Structure Understanding of Newspaper Images

arXiv:2607. 15082v1 Announce Type: cross Abstract: Understanding newspaper images remains a challenging task due to their complex, nested hierarchical structures and dense, heterogeneous layouts.

By William Moca\"er, Sol\`ene Tarride, Thomas Constum, Merveilles Agbeti-Messan, Tom Simon, Cl\'ement Chatelain, St\'ephane Nicolas, Pierrick Tranouez, S\'ebastien Cretin, Thierry Paquet

llms computer-vision benchmarks safety

arXiv AI

Jul 17

Mask-Aware Policy Gradients for Diffusion Language Models

arXiv:2607. 15200v1 Announce Type: cross Abstract: Reinforcement learning has proven effective for improving reasoning in large language models, but extending it to Masked Diffusion Language Models (MDLMs) remains challenging due to the intractability of the log-likelihood estimation.

By Haran Raajesh, Kulin Shah, Adam Klivans, Philipp Kr\"ahenb\"uhl

llms diffusion reinforcement-learning benchmarks

arXiv Machine Learning

Jul 17

A Weak Penalty Neural ODE for Learning Chaotic Dynamics from Noisy Time Series

arXiv:2511. 06609v4 Announce Type: replace Abstract: The accurate forecasting of complex, high-dimensional dynamical systems from observational data is a fundamental task across numerous scientific and engineering disciplines.

By Xuyang Li, John Harlim, Dibyajyoti Chakraborty, Romit Maulik

benchmarks

arXiv Machine Learning

Jul 17

The Effect of Stochasticity in Score-Based Diffusion Sampling: a KL Divergence Analysis

arXiv:2506. 11378v3 Announce Type: replace Abstract: Sampling in score-based diffusion models can be performed by solving either a reverse-time stochastic differential equation (SDE) parameterized by an arbitrary stochasticity function or a probability flow ODE, corresponding to setting this stochasticity function to zero.

By Bernardo P. Schaeffer, Ricardo M. S. Rosa, Glauco Valle

diffusion benchmarks

arXiv Machine Learning

Jul 17

Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization

arXiv:2605. 21751v2 Announce Type: replace Abstract: Text-to-optimization requires two separable capabilities: modeling -- choosing the right optimization structure -- and binding -- grounding every coefficient, index, and parameter in the concrete problem data.

By Zhiqi Gao, Albert Ge, Alexander Berenbeim, Nathaniel D. Bastian, Frederic Sala

reinforcement-learning fine-tuning benchmarks

arXiv AI

Jul 17

Quality-Aware Robust Multi-View Clustering for Heterogeneous Observation Noise

arXiv:2602. 22568v2 Announce Type: replace-cross Abstract: Deep multi-view clustering has achieved remarkable progress but remains vulnerable to complex noise in real-world applications.

By Peihan Wu, Guanjie Cheng, Yufei Tong, Meng Xi, Shuiguang Deng

benchmarks

arXiv Machine Learning

Jul 17

GAttNHP: Group Attention Neural Hawkes Process for Extrapolation Reasoning in Temporal Knowledge Graphs

arXiv:2607. 14733v1 Announce Type: new Abstract: Temporal Knowledge Graphs (TKGs) record how facts evolve over time, but forecasting future events on a TKG remains difficult for three reasons: (i) long-range temporal dependencies are hard to encode; (ii) events on different chains mutually excite or inhibit one another in ways that snapshot-level models cannot express; and (iii) inter-arrival times are heavy-tailed and statistically sparse, so deterministic time predictors are unreliable.

By Xiangni Tian, Kaixian Yu, Runpeng Dai, Niansheng Tang, Hongtu Zhu

benchmarks

arXiv AI

Jul 17

LBA: Textual Hard-Label Adversarial Attack under Low Query Budgets

arXiv:2607. 14101v1 Announce Type: cross Abstract: Generating high-quality adversarial texts with low query budgets remains a challenging problem in the hard-label scenario.

By Shixin Guo, Ming Zhong, Xuhong Zhang, Dandan Zhao, Zhe Wang, Bo Zhang, Shouling Ji, Hao Peng

llms benchmarks safety

arXiv AI

Jul 17

SmartRAG: Native Graph-Based RAG for Mobile Device

arXiv:2607. 14661v1 Announce Type: new Abstract: Deploying large language models (LLMs) as personal assistants on mobile devices demands privacy, low latency, and offline availability, yet the computational cost of giant models clashes with strict edge-hardware budgets.

By Zhihan Jiang, Meng Li, Shenghao Liu, Keran Li, Ruiben Zhou, Xianjun Deng, Shuai Wang, Haipeng Dai

llms rag efficiency benchmarks safety

arXiv AI

Jul 17

Mixtures of SubExperts for Large Language Continual Learning

arXiv:2511. 06237v2 Announce Type: replace-cross Abstract: Enabling lifelong learning in LLMs demands resolving the stability-plasticity dilemma (i.

By Haeyong Kang, Hee Suk Yoon, Dahua Feng, Chang D. Yoo

llms fine-tuning efficiency benchmarks

arXiv AI

Jul 17

MCPEvol-Bench: Benchmarking LLM Agent Performance Across Dynamic Evolutions of MCP Servers

arXiv:2607. 14642v1 Announce Type: new Abstract: As Model Context Protocol (MCP) servers emerge as the core infrastructure for connecting LLMs with external tools, existing benchmarks leverage real-world MCP servers to evaluate LLM agents' tool-using capabilities.

By Huanxi Liu, Kun Hu, Jiaqi Liao, Qiang Wang, Pengfei Qian, YuanZhao Zhai, Dawei Feng, Bo Ding, Huaimin Wang

llms agents benchmarks

arXiv Machine Learning

Jul 17

Lyapunov Guidance: A Unified Framework for Stabilizing Generative Flows

arXiv:2607. 14272v1 Announce Type: new Abstract: Flow matching has emerged as an effective framework for learning complex data distributions, but adapting pretrained flow models to new tasks often requires computationally expensive retraining.

By Jingdong Zhang, Xinze Li, Yize Jiang, Luan Yang, Minkai Xu, Junhong Liu

diffusion reinforcement-learning benchmarks

arXiv AI

Jul 17

VOiLA: Vectorized Online Planning with Learned Diffusion Models for POMDP Agents

arXiv:2606. 19729v2 Announce Type: replace-cross Abstract: Planning under uncertainty is an essential capability for autonomous robots.

By Marcus Hoerger, Rishikesh Joshi, Rahul Shome, Ian Manchester, Hanna Kurniawati

agents diffusion reinforcement-learning robotics efficiency benchmarks

Get the daily briefing