arXiv Computation and Language

ARE-bench: Benchmarking Agents' Research Ability in Real-World Researcher Sessions and Workspaces

ARE‑bench is a new benchmark that tests research agents on real‑world researcher sessions and workspaces, focusing on their ability to advance ongoing projects while respecting prior decisions and existing workspace states. It contains 75 tasks derived from actual researcher projects, covering seven research stages, 33 requirements, and 150 sub‑requirements, with a total of 1,173 binary checklist items. The benchmark shows that even the best frontier models only pass about two‑thirds of the tasks when full contextual requirements are considered, highlighting the difficulty of evaluating agents in realistic settings.

arXiv AI
Nov 5

SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

SlackBench is a new benchmark that evaluates AI agents on realistic collaborative research and engineering workflows, requiring joint reasoning over workplace communication and evolving repository state. The benchmark is built from privacy‑preserving reenactments of real academic projects, providing reproducible environments with multi‑party dialogue, code changes from GitHub, and Google Docs. It contains 140 queries covering heterogeneous‑source search, abstention, summarization, and inference over long conversations, and shows that current models perform well below the ceiling, indicating significant room for improvement.

By Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis
arXiv AI
Oct 15

Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fi...

By Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Chengsong Huang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ergan Shang, Fuxiao Liu, Tong Zheng, Ninghao Liu, Lichao Sun, Haitao Mi, Leowei Liang
arXiv AI
Nov 6

Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

The paper discusses the problem of benchmark maintenance debt in evaluating coding agents, where benchmarks become outdated as models improve and infrastructure changes. It introduces a continuous‑validation workflow that incorporates community feedback, automated checks, and human repairs, applied to Terminal‑Bench 2.0. The audit uncovered issues in about 30% of the benchmark, leading to a new version (Terminal‑Bench 2.1) that improved accuracy by over 10 percentage points and altered leaderboard positions, while also highlighting the importance of validating resource budgets alongside task artifacts.

By E. Kelly Buchanan, Crystal Zhou, Scott W. Linderman, Christopher R\'e
arXiv AI
Oct 2

Agents Are Systems, Not Models: Rethinking Agentic Evaluation

The paper argues that evaluating agents as fixed models is insufficient, proposing instead to treat them as configurable systems. Using a new benchmark of four scientific tasks, the authors analyze how five configuration aspects—task information, reasoning, self‑verification, time budget, and backbone model—affect performance, noting that about 54% of outcome variance arises from run‑to‑run differences even with the same settings. The study finds that providing more task information has the strongest impact, while interactions among settings (e.g., extra time only helps with adequate information or model capability) and the choice of verification tools significantly shape agent behavior.

By Luis Wiedmann, Leander Girrbach, Cordelia Schmid, Zeynep Akata
arXiv Computation and Language
Oct 15

Interlock: From Local Correctness to Task Completion in Long-Horizon Agents

Interlock is a framework that transforms long‑horizon tasks into typed completion graphs, linking evidence, decisions, and required effects through grounding, coupling, and realization edges. It compiles these graphs into a resettable workspace, a public request, and an evaluation rubric, and uses source‑coverage, public‑input, and agent‑as‑judge reviews to ensure that all mandatory requirements are met. Fine‑tuning Qwen3.6 backbones on Interlock‑generated trajectories improves pass rates on AutomationBench, JobBench, and Harvey LAB, showing that the synthesized experience benefits external task environments.

By Linji Hao, Yanji He, Zihan Wang, Hansi Yang, Jiayuan Zhang, Hongyan Hao, Qi Gu, Yan Feng, Han Wu
arXiv Computation and Language
Oct 26

SCI-AGENT-EXAM: Benchmarking Agents on Scientific Research Tasks

arXiv:2610.32441v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly capable of supporting scientific workflows, but their capabilities across diverse research activitie...

By Tianyun Zhong, Wei Wang, Jialin Li, Jianhong Tu, Liyao Li, Chuqiao Kuang, Tianshi Zheng, Songchi Zhou, Weiqi Zhai, Qile Jin, Jiajia Li, Jiasheng Zheng, Boxi Cao, Yaojie Lu, Hongyu Lin, Shiwei Ye, Bing Zhao, Hu Wei, Lin Qu, Xianpei Han
arXiv AI
Aug 19

StagedWorkspace: A Versioned Workspace for Knowledge-Work Agents

StagedWorkspace is a versioned workspace designed for knowledge‑work AI agents, ensuring that every parsed view, native file edit, and review diff is explicitly tied to a specific version of the workspace state. By binding parsed records and review diffs to content hashes of native files, the system improves performance on tasks such as OfficeQA and APEX‑Agents, achieving higher pass rates and rubric scores compared to single‑view approaches. The study demonstrates that providing dual parsed/native access and visible diffs enhances agent performance, highlighting workspace state as a key experimental variable for future benchmarks.

By Yining Hua, Hongbin Na, Yifan Zhou, Akshay Kalose, Cyrus Ayubcha, Levi Lian
arXiv AI
Oct 26

Benchmarks Are Surprisingly Compaction-Friendly

The paper investigates how context compaction—losing information to free up a language model’s context window—affects performance on long‑horizon agentic tasks. Using the OpenCode scaffold and five compaction strategies, the authors evaluate three frontier models on PaperBench, ProgramBench, and PostTrainBench, finding that compaction thresholds largely do not hurt performance for 1M‑context models, but can degrade GPT‑5.3‑Codex on certain benchmarks. The study also shows that compaction can silently drop critical task constraints, especially on ProgramBench and real user‑agent sessions from SWE‑chat, yet current benchmarks remain largely compaction‑friendly because objectives are fixed and progress is stored externally.

By Jeanne Salle, Hardik Bhatnagar, Matthias Bethge, Sahar Abdelnabi, Maksym Andriushchenko
Hugging Face Trending Papers
Nov 5

Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks

The paper discusses the problem of benchmark maintenance debt in evaluating coding agents, where benchmarks become outdated as models improve and infrastructure changes. It introduces a continuous‑validation workflow that incorporates community feedback, automated checks, and human repairs, applied to Terminal‑Bench 2.0, resulting in a revised version (Terminal‑Bench 2.1) that improves accuracy by over 10 percentage points and shifts leaderboard positions. The study also shows that resource allocation, such as time limits, can significantly alter agent performance, underscoring the need to validate both tasks and resource budgets.