arXiv Computation and Language By Jiahang Lin, Wanying Ren, Wanghan Xu, Huayu Sha, Yang Shu, Zhonghang Lu, Chengjun Pan, Yi Dong, Shichun Liu, Yuyang Hu, Shihan Dou, Honglin Guo, Huijie Lv, Pujun Zheng, Zhe Sun, Yuankai Ying, Yiding Wang, Jiahao Chen, Zhixiong Yang, Dingbo Yuan, Yicheng Zou, Zhecan Wang, Guoxiu He, Xiaoyu Xing, Qi Zhang, Tao Gui, Xuanjing Huang

ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces

Read the original on arXiv Computation and Language →

ARE‑bench is a new benchmark that tests research agents on real‑world researcher sessions and workspaces, focusing on their ability to advance ongoing projects while respecting prior decisions and existing workspace states. It contains 75 tasks derived from actual researcher projects, covering seven research stages, 33 requirements, and 150 sub‑requirements, with a total of 1,173 binary checklist items. The benchmark shows that even the best frontier models only pass about two‑thirds of the tasks when full contextual requirements are considered, highlighting the difficulty of evaluating agents in realistic settings.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv AI
Nov 5

SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

SlackBench is a new benchmark that evaluates AI agents on realistic collaborative research and engineering workflows, requiring joint reasoning over workplace communication and evolving repository state. The benchmark is built from privacy‑preserving reenactments of real academic projects, providing reproducible environments with multi‑party dialogue, code changes from GitHub, and Google Docs. It contains 140 queries covering heterogeneous‑source search, abstention, summarization, and inference over long conversations, and shows that current models perform well below the ceiling, indicating significant room for improvement.

By Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis
arXiv AI
Oct 15

Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fi...

By Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Chengsong Huang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ergan Shang, Fuxiao Liu, Tong Zheng, Ninghao Liu, Lichao Sun, Haitao Mi, Leowei Liang
arXiv AI
Nov 6

Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

The paper discusses the problem of benchmark maintenance debt in evaluating coding agents, where benchmarks become outdated as models improve and infrastructure changes. It introduces a continuous‑validation workflow that incorporates community feedback, automated checks, and human repairs, applied to Terminal‑Bench 2.0. The audit uncovered issues in about 30% of the benchmark, leading to a new version (Terminal‑Bench 2.1) that improved accuracy by over 10 percentage points and altered leaderboard positions, while also highlighting the importance of validating resource budgets alongside task artifacts.

By E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher R\'e