arXiv AI August 3, 2026 By Ismayil Ismayilov, Atakan Kara, Kaan Oktay

DungeonBench: A Benchmark for Rules-Rich Tactical Reasoning in Dungeons & Dragons Combat

arXiv:2607. 29577v1 Announce Type: new Abstract: Games and simulators make valuable benchmarks by turning decisions into measurable outcomes, but many current suites under-test rules-rich tactical reasoning: the ability to choose well when geometry, timing, resources, objectives, and rule interactions all matter at once.

Summary generated by The Flow from the publisher's feed. The full article lives at arXiv AI.

arXiv AI

Jul 3

AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents

arXiv:2607. 02255v1 Announce Type: new Abstract: Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see.

By Xiangchen Cheng, Yunwei Jiang, Jianwen Sun, Zizhen Li, Chuanhao Li, Xiangcheng Cao, Yihao Liu, Fanrui Zhang, Li Jin, Kaipeng Zhang

llms agents benchmarks

arXiv AI

Jun 18

RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models

arXiv:2606. 18950v1 Announce Type: new Abstract: Modern Vision-Language Models (VLMs) often struggle with strategic reasoning, i.

By San Kim, Daechul Ahn, Reokyoung Kim, Hyeonbeom Choi, Seungyeon Jwa, Jonghyun Choi

llms agents multimodal benchmarks

arXiv Machine Learning

Aug 4

AdvPlan-Bench: Adversarial Evaluation of Structured Plan-Generation Agents

arXiv:2608. 00832v1 Announce Type: new Abstract: Structured plan-generation agents are often evaluated as if a plan has quality in isolation, yet many realistic planning tasks require asking how a candidate behaves when another agent can search for responses.

By Alina Kapanova, Arun Kanhai, Natan Vidra, Spurthi Setty

llms agents benchmarks safety

arXiv AI

Aug 11

DSLE: A Learning Environment for Dark Souls Boss Encounters

arXiv:2608. 09902v1 Announce Type: new Abstract: We introduce the Dark Souls Learning Environment (DSLE), a containerized platform that presents all 22 boss encounters of Dark Souls: Remastered as game-playing agent benchmarks through a Gymnasium-style interface.

By Derin Gezgin, Jim O'Connor, Tanner Goodwin, Gary B. Parker

agents reinforcement-learning benchmarks

arXiv AI

Jul 1

An Executable Benchmarking Suite for Tool-Using Agents

arXiv:2605. 11030v2 Announce Type: replace-cross Abstract: Closed-loop tool-using agents are increasingly evaluated in executable web, code, and micro-task environments, but benchmark reports often conflate workloads, action-generating drivers, and the evidence admitted for systems-facing claims.

By Zhiqing Zhong, Zhijing Ye, Jiamin Wang, Xiaodong Yu

agents benchmarks

arXiv Machine Learning

Jun 2

ATLAS: Agentic Test-time Learning-to-Allocate Scaling

arXiv:2606. 01667v1 Announce Type: new Abstract: Test-time scaling has become a major way to improve large language model reasoning, but its orchestration has remained designer-engineered: a fixed sample budget, a fixed refinement loop, a fixed scoring rule, or a fixed search policy decides how compute is spent, leaving the model in charge of solving but not of orchestration.

By Peijia Qin, Qi Cao, Pengtao Xie

llms agents nlp multimodal benchmarks

Related stories

AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents

RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models

AdvPlan-Bench: Adversarial Evaluation of Structured Plan-Generation Agents

DSLE: A Learning Environment for Dark Souls Boss Encounters

An Executable Benchmarking Suite for Tool-Using Agents

ATLAS: Agentic Test-time Learning-to-Allocate Scaling