arXiv AI
Jun 10
Flaws in the LLM Automation Narrative
arXiv:2606. 11166v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly described as performing at the level of human experts on knowledge economy tasks.
Mind the Gap: Can Frontier LLMs Pass a Standardized Office Proficiency Exam?
arXiv:2606. 10956v1 Announce Type: new Abstract: The deployment of Large Language Model (LLM) agents for computer automation is accelerating, yet their ability to navigate complex, professional-grade productivity software is largely untested.
Related stories
arXiv AI
Jul 15
SheetMind: An End-to-End LLM-Powered Multi-Agent Framework for Spreadsheet Automation
arXiv:2506. 12339v2 Announce Type: replace-cross Abstract: We present SheetMind, a modular multi-agent framework powered by large language models (LLMs) for spreadsheet automation via natural language instructions.
Hugging Face Trending Papers
Aug 6
Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents
Large language models (LLMs) increasingly support complex professional tasks, yet their capabilities in rule-intensive document review remain insufficiently evaluated. National standard documents, such as China GB/T standards, offer a representative testbed: they are lengthy, highly structured, and governed by explicit rules for scope, terminology, normative wording, and cross-section consistency.
arXiv AI
Jul 3
Office Comprehension Benchmark
arXiv:2607. 01245v1 Announce Type: cross Abstract: We introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension over native file formats (.
arXiv AI
Aug 20
CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks
CentaurBench introduces a benchmark that evaluates large language models (LLMs) on both automating and augmenting the performance of a lower-capacity worker model across seven real-world tasks. The study finds that rankings for automation and augmentation are only modestly correlated, with automation winners often losing in augmentation scenarios. Assistance from higher-capacity models does not consistently improve outcomes, and in some cases the unaided worker performs best.
arXiv AI
Aug 18
PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflows
arXiv:2607. 06008v3 Announce Type: replace Abstract: While Large Language Model (LLM) agents excel at monolingual long-horizon planning and tool use, enterprise workflows inherently require processing multilingual resources across extended trajectories.
arXiv AI
Jun 18
CAPRA: Scaling Feedback on Software Architecture Deliverables with a Multi-Agent LLM System
arXiv:2606. 18976v1 Announce Type: cross Abstract: Automated assessment in software engineering education has advanced significantly for code grading and essay scoring.
arXiv Machine Learning
Jun 8
Elmes*: Automated Construction of Fine-Grained Evaluation Rubrics for Large Language Models in Long-Tail Educational Scenarios
arXiv:2606. 06546v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for education requires measuring how models teach, not only what they know.
Hugging Face Trending Papers
Jun 29
SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows
Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making. However, most existing spreadsheet benchmarks evaluate isolated operations such as single-formula generation or local cell edits, and therefore fail to capture end-to-end workflows in realistic business settings.
arXiv AI
Jul 31
IDP AutoOpt: Agent-Driven Optimization of Document Processing Pipeline Configurations
arXiv:2607. 26075v1 Announce Type: cross Abstract: We present IDP AutoOpt, an autonomous LLM agent that discovers high-performing configurations for intelligent document processing (IDP) pipelines.
arXiv AI
Jun 30
SpreadsheetBench 2: Evaluating Agents on End-to-End Business Spreadsheet Workflows
arXiv:2606. 29955v1 Announce Type: cross Abstract: Spreadsheets are widely used for business analysis, financial modeling, reporting, and decision-making.
arXiv AI
Sep 2
UniACE: A Unified Framework for Evaluating LLM Agentic Capabilities
UniACE is a unified framework that standardizes the evaluation of large language model (LLM) agents by representing each benchmark as an instruction–tool–environment triplet and running models through a shared, task‑agnostic harness in isolated runtimes. It preserves native success criteria, offers an offline mode for dynamic‑resource tasks, and standardizes efficiency metrics, execution records, and failure attribution. Applying UniACE to 7 benchmarks across 24 domains and 15 models revealed significant score shifts, ranking reversals, and sensitivity to evidence representation, highlighting the impact of evaluation configuration on reported agent performance.