arXiv AI September 2, 2026 By Jiaze Li, Aocheng Shen, Bing Liu, Boyu Zhang, Xiaoxuan Fan, Qiankun Zhang, Xianjun Deng

InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information

InteractBench is a new benchmark that evaluates large language models on interactive competitive programming problems, where key information is revealed only through queries to an interactor. It contains 322 curated problems from Codeforces, AtCoder, IOI, and ICPC, each with executable local interactors for offline testing. The study finds that even top reasoning models struggle with these tasks, highlighting gaps in information acquisition, state tracking, and protocol adherence.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI

Sep 2

CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?

arXiv:2609.01600v1 Announce Type: cross Abstract: Dynamic agent harnesses let language models change the software that shapes their own execution. This flexibility brings a new reasoning burden: a lo...

By Damien Sileo, Dimitri Kachler

llms agents benchmarks

arXiv AI

Sep 24

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

The paper introduces SWE-Flux, a repository‑level benchmark designed to test large language models’ ability to reason about runtime behavior. It contains 480 execution‑grounded instances from 12 real Python repositories, with gold answers automatically harvested from instrumented test executions. Evaluation of five LLMs shows the task remains difficult, with the best model achieving only 37% accuracy, and the benchmark can generate challenging variants through input perturbation.

By Hamed Taherkhani, Mohammad Abdollahi, Melika Sepidband, Hridya Dhulipala, Tien N. Nguyen, Hadi Hemmati

llms benchmarks

arXiv AI

Jun 3

Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphs

arXiv:2606. 03705v1 Announce Type: new Abstract: Knowledge Graphs (KGs) are widely used to mitigate the limitations of Large Language Models (LLMs), such as outdated knowledge and hallucinations.

By Weiwei Ding, Zixuan Li, Long Bai, Zhuo Chen, Kun Su, Fei Wang, Xiaolong Jin, Jin Zhang, Jiafeng Guo, Xueqi Cheng

llms benchmarks safety

arXiv AI

Jun 2

Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games

arXiv:2606. 00103v1 Announce Type: new Abstract: We introduce a multi-turn interactive framework for reasoning evaluation that treats reasoning as active evidence acquisition and belief updating.

By Mingyuan Fan, Weiguang Han, Daixin Wang, Cen Chen, Zhiqiang Zhang, Jun Zhou

llms benchmarks

arXiv AI

Aug 13

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

arXiv:2608. 12282v1 Announce Type: new Abstract: Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation.

By Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder, Siyu Huo, Raavi Gupta, Abhinav Jain, Praveen Venkateswaran, Abdulhamid Adebayo, Danish Contractor

agents benchmarks

arXiv AI

Sep 15

ZEBRAARENA: A Diagnostic Simulation Environment for Studying Reasoning-Action Coupling in Tool-Augmented LLMs

arXiv:2603.18614v2 Announce Type: replace Abstract: Tool-augmented large language models (LLMs) must tightly couple multi-step reasoning with external actions, yet existing benchmarks often confound...

By Wanjia Zhao, Ludwig Schmidt, Yejin Choi, James Zou, Vidhisha Balachandran, Lingjiao Chen

llms agents benchmarks

One email a morning, machine-written

Related stories

CordisBench: Can Language Models Reason About Component Lifecycles in Dynamic Agent Harnesses?

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

Code-on-Graph: Iterative Programmatic Reasoning via Large Language Models on Knowledge Graphs

Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games

VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies

ZEBRAARENA: A Diagnostic Simulation Environment for Studying Reasoning-Action Coupling in Tool-Augmented LLMs