arXiv AI
Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Sep 11
The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents
The Era by Eon Benchmark is a new dataset for evaluating large language model agents that interact with enterprise tools. It constructs a complete fictional company with product simulators, internal databases, and benchmark questions, all generated from a shared entity graph to ensure consistency. Exact answer keys are computed from the generated records, allowing precise grading and validation of realism and adversarial robustness across 23 simulated companies.
arXiv AI
Aug 5
GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks
arXiv:2608. 03764v1 Announce Type: new Abstract: Agent self-evolution updates an agent's persistent state from prior experience and reuses it to solve related tasks more effectively.
arXiv AI
Sep 17
ERPBench: A State-Grounded Evaluation Paradigm for Computer-Use Agents in Enterprise Software
ERPBench introduces a new evaluation paradigm for computer-use agents that operate via screenshots and simulated actions, focusing on enterprise software such as ERP systems. The benchmark tests agents on a live, reproducible ERP platform and scores tasks against ground-truth database values, highlighting challenges like dense interfaces, multi-step interactions, and persistent record errors. Experiments with six agents show that strong general GUI performance does not translate to reliable enterprise outcomes, with many agents frequently saving incorrect data.
arXiv AI
Aug 28
GROUND: Reducing Hallucinations in LLM-Based Enterprise Analytics Through Governed Semantic Definitions
The paper introduces GROUND, a framework that limits large language model (LLM) analytics to a governed semantic layer for enterprise data warehouses. GROUND supplies approved metrics, dimensions, join paths, filters, and security rules, then validates generated SQL against these constraints before execution, retrying or abstaining on violations. In benchmarks, GROUND eliminates hallucinations across all evaluated categories and prevents row‑level security breaches, outperforming schema‑only, schema‑RAG, and semantic‑only approaches.
arXiv AI
Sep 10
DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents
DI-Bench is a pipeline that automatically creates realistic data intelligence benchmarks for enterprise agents by linking data tables, dimensions, metrics, and documents into an artifact graph. It generates questions that combine structured data queries with knowledge retrieval, validates answers via query execution and LLM-generated questions, and has produced a 731-task benchmark covering knowledge retrieval, analytical computation, and rule‑grounded reasoning. Evaluation of four models on this benchmark shows that only 32% accuracy is achieved on computational tasks that involve business rules modifying the computation.
arXiv AI
Sep 7
ERPBench: Evaluating LLM Agents for Enterprise Decision-Making Across Competitive Market Ecologies
ERPBench is a benchmark that evaluates large language model agents in enterprise decision-making through a six‑round ERP simulation covering pricing, production, procurement, inventory, finance, and market competition. It tests the same 100 problems in two market ecologies—Solo, where agents compete against rule‑based opponents, and Arena, where six agents compete together—producing 1,200 model trajectories across 7,200 decision rounds. Results show that model performance varies by ecology, with DeepSeek best in Solo and Gemini best in Arena, and only 21 of 100 problems yield the same top performer across both settings.