MBABench-2.0 is a new benchmark that tests AI agents on building financial spreadsheets from scratch, featuring 101 original construction tasks that require complex modeling decisions, formatting, and layout design. Unlike previous fill‑in‑the‑blank tests, this benchmark evaluates spreadsheets on 129 quality criteria across 12 categories, addressing readability, maintainability, and auditability. Across 20 agents and 8 models, the best performer achieved only a 16% pass rate, highlighting a significant gap in current AI spreadsheet‑building capabilities.
arXiv:2605. 22664v2 Announce Type: replace Abstract: LLM agents are increasingly expected to carry out end-to-end workflows, producing complete artifacts from high-level user instructions.
Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making. However, most existing spreadsheet benchmarks evaluate isolated operations such as single-formula generation or local cell edits, and therefore fail to capture end-to-end workflows in realistic business settings.
arXiv:2606. 29955v1 Announce Type: cross Abstract: Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making.
Spreadsheets and tables are widely used representations for structured data analysis, but effective analysis still requires substantial manual effort and domain expertise. Recent large language model (LLM) agents can automate parts of this process, but they often provide limited transparency into intermediate decisions, rely on implicit assumptions, struggle with multi-table comparison, and repeat similar workflows without adapting to a user's preferences.
arXiv:2608. 07873v1 Announce Type: new Abstract: We introduce the workbook time machine, a pipeline that automatically creates benchmarks evaluating the ability of language models to create derived objects in spreadsheets (formulas, charts, pivot tables, and conditional formatting).
arXiv:2608. 16045v1 Announce Type: cross Abstract: LLM-based data-analysis tools are increasingly used to help users analyze messy spreadsheets and workbooks, from answering questions over uploaded files to generating code, summaries, and visualizations.
AgenticFinBench is a new benchmark comprising 230 real‑world financial tasks across eight categories, two languages, and four complexity levels, designed to capture the breadth and open‑ended nature of financial AI work. It introduces a reference‑free evaluation framework that uses question‑specific rubrics and a two‑stage claim audit to detect hallucinations, aligning assessment more closely with real‑world financial practice. Experiments on 18 agent configurations show that ignoring hallucinations inflates performance, that harness differences drive evidence grounding more than model differences, and that task complexity reduces quality scores, highlighting challenges in explaining financial impacts and comparing data consistently.
arXiv:2603. 10002v2 Announce Type: replace-cross Abstract: We consider the task of end-to-end spreadsheet generation, where language models produce spreadsheet artifacts to satisfy users' explicit and implicit constraints, specified in natural language.
AgentSWE is a new benchmark for agent software engineering that evaluates coding agents across three lifecycle stages: creating an agent from a natural‑language requirement, editing existing agent codebases, and optimizing toward a measurable target. The benchmark consists of 25 commission‑style tasks, where builders iterate on a few illustrative cases and are then tested on held‑out cases that cover the full requirement, with hard pass/fail gates or objective‑based grading. Experiments show that current coding agents fail to meet the requirements, revealing common failure modes such as overfitting, incomplete runtimes, and unsupported completion claims.
arXiv:2607. 19409v1 Announce Type: new Abstract: Recent advances in large language models have accelerated deployment of agentic systems in operational finance.
arXiv:2506. 12339v2 Announce Type: replace-cross Abstract: We present SheetMind, a modular multi-agent framework powered by large language models (LLMs) for spreadsheet automation via natural language instructions.
arXiv:2609.07603v1 Announce Type: new Abstract: Financial scenarios are diverse and complex, spanning varying data conditions, tool configurations, and workflows. Yet existing CUA, Computer-Using Age...