Benchmarks and evaluation

Leaderboards, eval harnesses and ablations — the contested business of deciding which model is actually better.

16,906 stories · RSS feed

arXiv AI

Jul 13

ReinforceGen: Hybrid Skill Policies with Automated Data Generation and Reinforcement Learning

arXiv:2512. 16861v2 Announce Type: replace-cross Abstract: Long-horizon manipulation has been a long-standing challenge in the robotics community.

By Zihan Zhou, Animesh Garg, Ajay Mandlekar, Caelan Garrett

reinforcement-learning robotics fine-tuning benchmarks

arXiv AI

Jul 13

Transformer-Empowered Actor-Critic Reinforcement Learning for Sequence-Aware Service Function Chain Partitioning

arXiv:2504. 18902v3 Announce Type: replace-cross Abstract: In the forthcoming era of 6G networks, characterized by unprecedented data rates, ultra-low latency, and ubiquitous connectivity, effective management of Virtualized Network Functions (VNFs) is essential.

By Cyril Shih-Huan Hsu, Anestis Dalgkitsis, Paola Grosso, Chrysa Papagianni

llms reinforcement-learning benchmarks

arXiv AI

Jul 13

A Self-Evolving Agentic Framework for Metasurface Inverse Design

arXiv:2604. 01480v2 Announce Type: replace Abstract: Metasurface inverse design can realize complex optical functionality, but turning a target optical response into executable optimization code still requires substantial expertise in computational electromagnetics and solver-specific software engineering.

By Yi Huang, Bowen Zheng, Yunxi Dong, Hong Tang, Huan Zhao, S. M. Rakibul Hasan Shawon, Hualiang Zhang

agents benchmarks

arXiv AI

Jul 13

Inside the Skill Market: From Software Engineering Activities to Reusable Agent Skills

arXiv:2607. 09065v1 Announce Type: cross Abstract: Software engineering (abbrev.

By Jialun Cao, Xinru Yan, Songqiang Chen, Yaojie Lu, Zhongxin Liu, Shing-Chi Cheung

agents benchmarks safety

arXiv AI

Jul 13

Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models

arXiv:2607. 09654v1 Announce Type: cross Abstract: Vision language models (VLMs) have made remarkable progress in visual reasoning during the last decade.

By Shravan Murlidaran, Miguel P. Eckstein

llms computer-vision multimodal benchmarks safety

arXiv AI

Jul 13

OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documents

arXiv:2607. 09068v1 Announce Type: cross Abstract: Recent advancements in LVLMs necessitate robust benchmarks for complex, visually grounded reasoning.

By Yang Chen, Yunwen Li, Yufan Shen, Minghao Liu, Tianyu Zheng, Bin Fu, Qunshu Lin, Zhi Yu, Botian Shi

benchmarks

arXiv AI

Jul 13

Conceptual Networks for Cross-Linguistic Idiomatic Expressions:A Feature-Based Graph Approach

arXiv:2607. 09576v1 Announce Type: cross Abstract: We present an interpretable network-based framework for representing idiomatic and figurative meaning across eight typologically diverse languages, totaling 160 conventional expressions, the large majority of which are idiomatic.

By Kiran Pala, Punam Silu, Lixun Yu

llms rag benchmarks

arXiv AI

Jul 13

Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue

arXiv:2605. 12920v3 Announce Type: replace-cross Abstract: Effective collaboration between embodied agents requires more than acting in a shared environment; it demands communication grounded in each agent's evolving understanding of the world.

By Vardhan Dongre, Dilek Hakkani-T\"ur

llms agents robotics benchmarks safety

arXiv AI

Jul 13

What VGGT Knows About Overlap: Probing Geometric Foundation Models for Co-Visibility

arXiv:2607. 09503v1 Announce Type: cross Abstract: A fundamental challenge in 3D reconstruction and robotic localization is co-visibility: determining which image pairs share overlapping visible surfaces, particularly in scenarios with minimal overlap.

By Filippo Ziliotto, Luciano Serafini, Lamberto Ballan, Tommaso Campari

llms robotics benchmarks

arXiv Machine Learning

Jul 13

EvoLP: Self-Evolving Latency Predictor for Model Compression in Real-Time Edge Systems

arXiv:2607. 09063v1 Announce Type: new Abstract: Edge devices are increasingly utilized for deploying deep learning applications on embedded systems.

By Shuo Huai, Hao Kong, Shiqing Li, Xiangzhong Luo, Ravi Subramaniam, Christian Makaya, Qian Lin, Weichen Liu

efficiency benchmarks

arXiv Machine Learning

Jul 13

Pattern-Aware Graph Neural Networks for Handling Missing Data

arXiv:2607. 08915v1 Announce Type: new Abstract: Missing data is ubiquitous in real-world datasets.

By Minett Tran, Taehee Jeong

rag benchmarks

arXiv Machine Learning

Jul 13

Is data-efficient learning feasible with quantum models?

arXiv:2508. 19437v2 Announce Type: replace-cross Abstract: The importance of analyzing nontrivial datasets when testing quantum machine learning (QML) models is becoming increasingly prominent in literature, yet a cohesive framework for understanding dataset characteristics remains elusive.

By Alona Sakhnenko, Christian B. Mendl, Jeanette M. Lorenz

benchmarks safety

arXiv AI

Jul 13

Interference and Retention in Continual Learning

arXiv:2607. 09202v1 Announce Type: cross Abstract: Continual learning commonly relies on post-hoc mechanisms such as replay, elastic regularization, or distillation.

By Julius St\"ork

efficiency benchmarks

arXiv AI

Jul 13

Test-Time Scaling for Small VLMs on Multilingual Visual MCQ

arXiv:2607. 09438v1 Announce Type: cross Abstract: Test-time scaling (TTS) reliably improves reasoning in large language models, but whether it transfers to small open vision-language models remains unclear.

By Spiros Baxevanakis, Peng-Jian Yang

llms multimodal benchmarks

arXiv Machine Learning

Jul 13

Federated Low-Rank Koopman Learning for Multivariate Time-Series Anomaly Detection in IoT Systems

arXiv:2607. 08978v1 Announce Type: new Abstract: Distributed IoT systems generate multivariate time-series streams for monitoring physical assets, servers, and embedded sensing platforms.

By Tung-Anh Nguyen, Van-Phuc Bui, Anh Tuyen Le, Kim Hue Ta, Minh Thuy Le, J. Andrew Zhang, Xiaojing Huang

efficiency benchmarks

arXiv AI

Jul 13

Correlation-Aware Contextual Bandits with Surrogate Rewards for LLM Routing

arXiv:2607. 09015v1 Announce Type: cross Abstract: We study contextual bandit problems with correlated arms and access to surrogate reward signals produced by a machine learning model, motivated by applications such as large language model (LLM) routing.

By Ajay Narayanan Sridhar, Ronak Singh, Mehrdad Mahdavi, Vijaykrishnan Narayanan

llms reinforcement-learning benchmarks

arXiv Machine Learning

Jul 13

RaMark: Radioactive Watermarking for Generated Tabular Data

arXiv:2607. 09000v1 Announce Type: cross Abstract: Recent advances in generative modeling have made generated tabular data a practical solution for privacy-sensitive data sharing, where watermarking enables ownership verification.

By Xin Che, Lingyang Chu, Qiqi Zhang, Xinyu Ma, Xuan Luo, Jian Pei

benchmarks safety

arXiv AI

Jul 13

The LLMbda Calculus: AI Agents, Conversations, and Information Flow

arXiv:2602. 20064v2 Announce Type: replace-cross Abstract: Large language models are increasingly deployed as agents: they plan, call tools, read untrusted data, and act on the results.

By Zac Garby, Andrew D. Gordon, David Sands

llms agents benchmarks

arXiv AI

Jul 13

DaDaDa: A Dataset for Data Pricing in Data Marketplaces

arXiv:2607. 08785v1 Announce Type: cross Abstract: High-quality data drives machine learning advances across industries.

By Qiheng Sun, Hongwei Zhang, Junxu Liu, Xiaokai Mao, Jinfei Liu, Kui Ren, Haibo Hu

benchmarks

arXiv AI

Jul 13

Eluna: An Agentic LLM System for Automating Warehouse Operations with Reasoning and Task Execution

arXiv:2607. 08960v1 Announce Type: cross Abstract: Warehouse operations are governed by Standard Operating Procedures (SOPs) that encode complex, multi-system decision logic, which must be executed reliably under strict time constraints, yet LLM agents lack mechanisms to enforce procedural compliance and degrade under the context overload full SOP specifications introduce.

By Ning Liu, Kalle Kujanp\"a\"a, Zhaoxuan Zhu, P Aditya Sreekar, Kaiwen Liu, Chuanneng Sun, Jorge Marchena Menendez, Matthew Bales, Tianyu Yang, Shahnawaz Alam, Rose Yu, Baoyuan Liu, Kristina Klinkner, Shervin Malmasi

llms agents fine-tuning efficiency benchmarks

Get the daily briefing