Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

19,504 stories · RSS feed

arXiv AI

Jun 30

Managing the Human Fallback: Skill Investment Under Improving AI and Worker Mobility

arXiv:2606. 29111v1 Announce Type: new Abstract: When firms deploy autonomous AI, they must decide how much work to leave to the system and how much to keep workers engaged.

By Simrita Singh, Naireet Ghosh, Tinglong Dai

agents benchmarks

arXiv Machine Learning

Jun 30

A3M: Adaptive, Adversarial and Multi-Objective Learning for Strategic Bidding in Repeated Auctions

arXiv:2606. 28943v1 Announce Type: cross Abstract: Learning to bid in repeated multi-unit auctions with bandit feedback poses a fundamental challenge.

By Junhan Li, Yuxin Zhang, Haoran Wang, Minghao Chen

reinforcement-learning benchmarks safety

arXiv Machine Learning

Jun 30

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

arXiv:2606. 30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks.

By Mohit Raghavendra, Anisha Gunjal, Aakash Sabharwal, Yunzhong He

llms agents benchmarks

arXiv AI

Jun 30

Assessing the Business Process Modeling Competences of Large Language Models

arXiv:2601. 21787v2 Announce Type: replace-cross Abstract: The creation of Business Process Model and Notation (BPMN) models is a complex and time-consuming task requiring both domain knowledge and proficiency in modeling conventions.

By Chantale Lauer, Peter Pfeiffer, Alexander Rombach, Nijat Mehdiyev

llms fine-tuning benchmarks

arXiv AI

Jun 30

CMSL: Constructive Multi-Sequence Learning for Recommendation Systems

arXiv:2606. 28533v1 Announce Type: cross Abstract: Sequence learning has emerged as the promising paradigm in recommendation systems, surpassing traditional Deep Learning Recommendation Models (DLRM) by capturing the temporal nuances of user behavior.

By Zikun Cui, Renzhi Wu, Junjie Yang, Li Sheng, Jijie Wei, Linfeng Liu, Tai Guo, Tao Jia, Xiaodong Wang, Hong Li, Li Yu, Sri Reddy, Hong Yan

llms diffusion benchmarks

arXiv AI

Jun 30

FD$^2$: A Dedicated Framework for Fine-Grained Dataset Distillation

arXiv:2603. 25144v2 Announce Type: replace-cross Abstract: Dataset distillation (DD) compresses a large training set into a small synthetic set, reducing storage and training cost, and has shown strong results on general benchmarks.

By Hongxu Ma, Guang Li, Shijie Wang, Dongzhan Zhou, Baoli Sun, Takahiro Ogawa, Miki Haseyama, Zhihui Wang

efficiency benchmarks

arXiv AI

Jun 30

Pondering the Way: Spatial-perceiving World Action Model for Embodied Navigation

arXiv:2606. 29908v1 Announce Type: cross Abstract: Existing world model-based planners for visual navigation typically follow a verification-centric paradigm, decoupling goal intent from trajectory synthesis.

By Hong Chen, Daqi Liu, Zehan Zhang, Haiguang Wang, Tianhao Lu, Longfei Yan, Haiyang Sun, Fangzhen Li, Hongwei Xie, Bing Wang, Guang Chen, Hangjun Ye, Yihua Tan

robotics benchmarks safety

arXiv AI

Jun 30

DyGnROLE: Asymmetric Pretraining for Edge Classification on Dynamic Graphs

arXiv:2602. 23135v2 Announce Type: replace-cross Abstract: Edge classification on directed dynamic graphs requires modeling interactions between source and destination nodes exhibiting asymmetrical behavioral patterns and temporal dynamics.

By Tyler Bonnet, Marek Rei

llms rag benchmarks safety

arXiv Machine Learning

Jun 30

Two kinds of robustness are not the same: disentangling fault tolerance and low-SNR robustness in multi-domain event detection on real data

arXiv:2606. 29339v1 Announce Type: cross Abstract: Reliable event detection underpins induced-seismicity monitoring for Carbon dioxide Capture and Storage (CCS) and geothermal operations, distributed acoustic sensing (DAS), and industrial condition monitoring.

By Isao Kurosawa

llms benchmarks

arXiv AI

Jun 30

Geo-Expert: Towards Expert-Level Geological Reasoning via Parameter-Efficient Fine-Tuning

arXiv:2605. 24844v2 Announce Type: replace Abstract: While general-purpose Large Language Models (LLMs) applied to Geology often hallucinate when reasoning about subsurface structures and deep-time evolution, current AI in Earth sciences predominantly targets surface remote sensing and GIS.

By Chenyou Guo, Zongqi Liu, Yizhou Zhang, Zhaorui Jiang, Ze Liu

llms fine-tuning benchmarks

arXiv Machine Learning

Jun 30

Residual-Guided Dictionary Learning for Spectrally Accurate Koopman Approximation

arXiv:2606. 29083v1 Announce Type: cross Abstract: Koopman theory promises linear structure in nonlinear dynamics, but numerical Koopman spectra are easy to compute and hard to trust.

By George Coote, Matthew J. Colbrook

benchmarks

arXiv AI

Jun 30

Improvement of Robot's Simultaneous Localization and Mapping Using an Effective Transformation to Achieve Linear Model

arXiv:2606. 28475v1 Announce Type: cross Abstract: Nowadays mobile robots have wide engineering applications.

By Seyed Farzad Bahreinian, Maziar Palhang, Mohammad Reza Taban, Hasan Enami Eraghi

robotics benchmarks

arXiv AI

Jun 30

StarDojo: Benchmarking Open-Ended Behaviors of Agentic Multimodal LLMs in Production-Living Simulations with Stardew Valley

arXiv:2507. 07445v3 Announce Type: replace Abstract: Autonomous agents navigating human society must master both production activities and social interactions, yet existing benchmarks rarely evaluate these skills simultaneously.

By Weihao Tan, Changjiu Jiang, Yu Duan, Mingcong Lei, Jiageng Li, Yitian Hong, Xinrun Wang, Bo An

llms agents robotics multimodal benchmarks

arXiv Machine Learning

Jun 30

Memory-Managed Long-Context Attention: A Preliminary Study of Editable Request-Local Memory

arXiv:2606. 28876v1 Announce Type: cross Abstract: Long-context language models often conflate two different goals: compressing history into an efficient state, and maintaining reliable long-term memory.

By Junyi Zou, Avrova Donz

llms benchmarks

arXiv Machine Learning

Jun 30

Geometric Algebra Meets Cartesian Tensors: Higher-Order Equivariance for Interatomic Potentials

arXiv:2606. 29584v1 Announce Type: cross Abstract: $\mathrm{Cl}(3,0)$ interatomic potentials, despite their algebraic elegance, predict force magnitudes accurately but force directions poorly.

By Can Polat, Erchin Serpedin, Mustafa Kurban, Hasan Kurban

benchmarks

arXiv AI

Jun 30

ADEPT: An Entropy-Driven Dual-Strategy Agent for Interactive Video Retrieval

arXiv:2606. 28326v1 Announce Type: cross Abstract: This research aims to solve the challenge of video retrieval from massive datasets, caused by ambiguous user queries.

By Ke Chen, Shengyuan Han, Yongfeng Huang, Yujin Zhu, Jingwei Xiong, Liang Xu, Jundong Liu

llms agents benchmarks

arXiv AI

Jun 30

GraphChase: A Platform and Benchmark for Urban Network Security Games

arXiv:2501. 17559v2 Announce Type: replace Abstract: After the achievement of solving two-player zero-sum games, more AI researchers focus on solving multiplayer games.

By Shuxin Zhuang, Shuxin Li, Tianji Yang, Muheng Li, Xianjie Shi, Bo An, Youzhi Zhang

robotics benchmarks

arXiv AI

Jun 30

HippoSpark: An On-Demand Experience System for LLM Reasoning

arXiv:2606. 29929v1 Announce Type: new Abstract: Distilling historical trajectories into reusable experience to enhance future problem-solving has become a focal point of recent LLM research.

By Jingyao Liu, Danling Meng, Chen Huang, Yukun Yan, Zhenghao Liu, Wenqiang Lei, See-Kiong Ng, Maosong Sun

llms benchmarks

arXiv AI

Jun 30

Redefining Maritime Anomaly Detection via Equation-Grounded Synthetic Anomalies

arXiv:2606. 29721v1 Announce Type: cross Abstract: Maritime anomaly detection is essential for ensuring maritime safety, security, and efficient traffic management at sea, with Automatic Identification System (AIS) data serving as a primary data source.

By Youngseok Hwang, Sungho Bae, Dohun Lee, Jaeeun Seo, Jeehong Kim, Wonhee Lee, Hyunwoo Park

llms benchmarks safety

arXiv Machine Learning

Jun 30

Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

arXiv:2606. 28433v1 Announce Type: new Abstract: One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for learning in deployment settings.

By Matthew Vandergrift, Esraa Elelimy, Martha White

agents reinforcement-learning benchmarks

Get the daily briefing