arXiv AI

DiG-bench: Discovery in Games

arXiv:2608. 12593v1 Announce Type: new Abstract: Discovery---formulating novel generalizations---is a central part of the scientific process.

arXiv AI

Jul 17

CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents

arXiv:2607. 14386v1 Announce Type: new Abstract: Data science tasks span from closed-ended information extraction to open-ended analysis, presenting significant challenges for automation.

By Maxime Heuillet, Sharadind Peddiraju

llms agents benchmarks

arXiv AI

Jun 4

The Meta-Agent Challenge: Are Current Agents Capable of Autonomous Agent Development?

arXiv:2606. 04455v1 Announce Type: new Abstract: Current AI benchmarks evaluate agents on task execution within human-designed workflows.

By Xinyu Lu, Tianshu Wang, Pengbo Wang, zujie wen, Zhiqiang Zhang, Jun Zhou, Boxi Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun

agents benchmarks safety

arXiv AI

Jul 14

FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insights

arXiv:2602. 02905v2 Announce Type: replace Abstract: Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge.

By Zhen Wang, Fan Bai, Zhongyan Luo, Jinyan Su, Kaiser Sun, Xinle Yu, Jieyuan Liu, Kun Zhou, Claire Cardie, Mark Dredze, Zhiting Hu, Eric P. Xing

llms agents benchmarks

arXiv AI

Jun 18

RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models

arXiv:2606. 18950v1 Announce Type: new Abstract: Modern Vision-Language Models (VLMs) often struggle with strategic reasoning, i.

By San Kim, Daechul Ahn, Reokyoung Kim, Hyeonbeom Choi, Seungyeon Jwa, Jonghyun Choi

llms agents multimodal benchmarks

arXiv AI

Jun 12

Benchmarking AI Agents for Addressing Scientific Challenges Across Scales

arXiv:2606. 12736v1 Announce Type: new Abstract: AI agents are increasingly being developed to accelerate scientific discovery, yet their practical capabilities in real research settings remain poorly understood.

By Tianyu Liu, Allen Xin Wang, Antonia Panescu, Lisa Xinyi Chen, Wenxin Long, Xinyu Wei, Yueqian Jing, Ziyao Zeng, Jihang Chen, Sihan Jiang, Ziqing Wang, Siyi Gu, Siyu Chen, Xinyang Hu, Haoran Shao, Leqi Xu, Wangjie Zheng, Zhiyuan Cao, Ada Fang, Botao Yu, Kunyang Sun, Rex Ying, Arman Cohan, Qingyu Chen, Lingzhou Xue, Kaize Ding, Yuanqi Du, Wengong Jin, Zhuoran Yang, Marinka Zitnik, James Zou, Hua Xu, Hongyu Zhao

agents benchmarks

arXiv AI

Jul 14

GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learning

arXiv:2604. 02721v2 Announce Type: replace Abstract: Competitive programming remains one of the last few human strongholds in coding against AI.

By DeepReinforce Team, Xiaoya Li, Guoyin Wang, Songqiao Su, Chris Shum, Jiwei Li

llms agents nlp reinforcement-learning

arXiv AI

Jun 9

OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics

arXiv:2606. 09826v1 Announce Type: cross Abstract: Vision-language model (VLM) agents are increasingly deployed in interactive game environments.

By Mingxian Lin, Shengju Qian, Yuqi Liu, Yi-Hua Huang, Yiyu Wang, Wei Huang, Yitang Li, Fan Zhang, Zeyu Hu, Lingting Zhu, Xin Wang, Xiaojuan Qi

llms agents multimodal benchmarks

arXiv Machine Learning

Jul 7

MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

arXiv:2605. 08678v3 Announce Type: replace Abstract: Modern AI progress has been driven by ML methods that are generalizable across settings and scalable to larger regimes.

By Bohan Lyu, Yucheng Yang, Siqiao Huang, Jiaru Zhang, Qixin Xu, Xinghan Li, Xinyang Han, Yicheng Zhang, Huaqing Zhang, Runhan Huang, Kaicheng Yang, Zitao Chen, Wentao Guo, Junlin Yang, Xinyue Ai, Wenhao Chai, Yadi Cao, Ziran Yang, Kun Wang, Dapeng Jiang, Huan-ang Gao, Shange Tang, Chengshuai Shi, Simon S. Du, Max Simchowitz, Jiantao Jiao, Dawn Song, Chi Jin

llms agents benchmarks

arXiv AI

Jul 9

Reward-Adaptive Iterative Discovery: A Case Study on Automated Game Testing for NHL26

arXiv:2607. 07498v1 Announce Type: cross Abstract: Testing is a major effort for the gaming industry, requiring a significant part of development budget and people power.

By Florian Fuchs, Jessy Gosselin-Grant, Boris Skuin, Michele Petteni, Alessandro Sestini, Joakim Bergdahl, Amir Baghi, Linus Gissl\'en

agents reinforcement-learning

arXiv AI

6d ago

AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research

arXiv:2608. 11216v1 Announce Type: new Abstract: World modeling is an unsettled field: architectures, training objectives, and state representations interact in complex ways, and no single recipe dominates across environments.

By Marjan Moodi, Xuankang Zhu, Fernando De Mesentier Silva, Harold Chaput, Mohammad Reza Taesiri

llms agents benchmarks

arXiv AI

Jul 15

From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World

arXiv:2605. 10834v2 Announce Type: replace Abstract: AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perform best in real-world targets.

By Pedro Conde, Henrique Branquinho, Valerio Mazzone, Bruno Mendes, Andr\'e Baptista, Nuno Moniz

llms agents benchmarks

arXiv AI

Aug 5

Is Inter-Seed Cross-Play Enough? Evaluating the Robustness of Zero-Shot Coordination Algorithms to Implementation Details

arXiv:2608. 03644v1 Announce Type: new Abstract: AI agents deployed in real-world settings must be capable of coordinating with humans and other AI agents they have not encountered before.

By Maksymilian Wolski, Nicholas Hoernle, Johannes Forkel, Jakob Foerster

agents reinforcement-learning

Related stories