Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

28,306 stories · RSS feed

arXiv AI
Sep 24

OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents

arXiv:2609.23407v2 Announce Type: replace-cross Abstract: Humans can effortlessly localize the direction of a sound source and integrate it with visual cues for reasoning, yet this remains challengin...

By Ruixun Liu, Yuxuan Wang, Jiacheng Xie, Yuhuan You, Donghua Cai, Junming Lin, Xiong-Hui Chen, Zhifang Guo, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Yiwu Zhong
arXiv Machine Learning
Sep 24

SoLiD26: A First Principles Solid-Liquid Interface Dataset for Machine-learned Interatomic Potentials

SoLiD26 is a curated dataset of 15.4 million first‑principles atomic structures for solid‑liquid interfaces, comprising up to 576 atoms and 15 chemical elements. The data were generated from density functional theory calculations, mainly ab initio molecular dynamics, and include aqueous coinage metal interfaces, electrode‑electrolyte systems, and bulk references. Each record contains atomic species, positions, cell parameters, periodic boundary conditions, potential energy, and atomic forces, all computed with VASP using the PBE functional and D3 dispersion corrections.

By Jonas Busk, Emil J. P. Frost, Yogeshwaran Krishnan, Henrik H. Kristoffersen, August E. G. Mikkelsen, Xueping Qin, Xin Yang, Heine A. Hansen, Arghya Bhowmik, Tejs Vegge
arXiv AI
Sep 24

Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse

The paper audits silent failures in agent-to-tool interactions within the ToolUniverse environment, focusing on 15 scientific tools. It identifies 91 failures—primarily missing data or inconsistent search/filtering—occurring mainly in the API and wrapper layers, and shows how these silent failures can propagate downstream into seemingly valid outputs. The authors propose contextual reliability and recommend testing, disclosure, monitoring, and measurement strategies for the agent-tool pipeline.

By Shreya Gopalan, Devansh Singh, Sundaraparipurnan Narayanan