← Back to all news
arXiv AI By Mingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis

SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

Read the original on arXiv AI →

SlackBench is a new benchmark that evaluates AI agents on realistic collaborative research and engineering workflows, requiring joint reasoning over workplace communication and evolving repository state. The benchmark is built from privacy‑preserving reenactments of real academic projects, providing reproducible environments with multi‑party dialogue, code changes from GitHub, and Google Docs. It contains 140 queries covering heterogeneous‑source search, abstention, summarization, and inference over long conversations, and shows that current models perform well below the ceiling, indicating significant room for improvement.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

Related stories

arXiv Computation and Language
Oct 28

Choosing Coding-Agent Benchmarks for Practice: Lessons from Task Demands and Agent Behaviour

The paper analyzes 2,487 tasks from five public coding‑agent benchmarks and 14,922 runs of six agent configurations, introducing the Spread‑Novelty‑Centrality (SNC) profile to describe how reference solutions are distributed, added, and the code’s dependencies, history, and complexity. It finds that every pair of benchmarks differs on at least two SNC axes, even when sharing the same task label, and that resolved runs tend to involve less distributed changes and lower centrality values, with patch sizes varying between Claude and Qwen configurations. The authors recommend that practitioners choose evaluation tasks based on change characteristics, inspect task descriptions, and assess patch scope alongside test outcomes to better align benchmarks with real‑world workloads.

By Radin Shayanfar, Keheliya Gallaba, Ahmed E. Hassan
llmsagentsbenchmarks
More like this →
arXiv AI
Oct 26

What Open-Source Projects Tell Their Coding Agents: A Census of AGENTS.md and CLAUDE.md

The study surveys popular, active GitHub repositories across thirteen languages, treating each repository’s AGENTS.md and CLAUDE.md files as instruction sets for coding agents. It finds that most repositories include such instructions, with CLAUDE.md often referencing AGENTS.md, and that these files contain a mix of project knowledge and workflow rules, including varying stances on AI attribution. The analysis reveals that more popular projects tend to have stricter pull‑request rules and AI contribution guidelines, and that LLM and agent projects are over‑represented among adopters.

By Pawel Zarzycki
llmsagents
More like this →