AvalancheBench is a multimodal enterprise analytics benchmark that tests data agents’ ability to recover a latent world from observed data, using 58 questions across text, images, and documents. It evaluates agents on analytical fidelity, monetary cost, and a new capability‑adjusted token cost (CATC) metric, revealing that misinterpreting requirements and misuse of semantic operators are the biggest sources of error. The benchmark highlights how errors can propagate through dependent questions, affecting overall performance.
arXiv:2610.34860v1 Announce Type: new Abstract: Much of the information needed for data analysis is buried in text columns: an analyst may infer customer frustration from complaint text and use it to...
EGI‑BENCH is a unified benchmark suite for evaluating large language model agents in enterprise settings, consolidating eight existing benchmarks (CRMARENA, HERB, PERSONABENCH, USERBENCH, MCP‑UNIVERSE, MCPEVAL, BENCHFORCE, LOCOBENCH) under a single command‑line interface with two runtime‑budgeted modes. It introduces the Enterprise General Intelligence (EGI) Score, a six‑dimensional framework (M1‑M6) that aggregates fine‑grained sub‑metrics from each benchmark rather than simple averages, providing a more nuanced view of model capabilities. The paper reports results for fourteen LLMs across GPT, Claude, and Gemini families, showing varied strengths across task outcome, planning, retrieval, tool actioning, safety, and engineering quality.
arXiv:2608. 03451v1 Announce Type: new Abstract: Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia.
arXiv:2606. 28370v1 Announce Type: cross Abstract: Enterprise business intelligence queries span structured warehouses and unstructured document repositories -- modalities with fundamentally different access methods, cost profiles, and correctness semantics.
arXiv:2605. 28787v2 Announce Type: replace-cross Abstract: In the era of autonomous agents, machine-actionable data is critical for data-driven workflows.
arXiv:2610.20959v2 Announce Type: replace-cross Abstract: Answering questions over heterogeneous data means combining evidence from files, databases, documents and the web. Data agents automate this,...
The paper introduces RAGFlow, a retrieval‑augmented generation framework that transforms an undisciplined query loop into a structured pipeline of five stages. It emphasizes knowledge compilation—enriching documents with navigation hierarchies, claim‑evidence pairs, entity graphs, timelines, mind maps, ontologies, wikis, and memory—rather than improving retrieval accuracy. The compiled context reduces query tokens by 91% when evidence suffices, cuts latency from 140 s to 30 s, and provides governance and integration benefits not captured by standard accuracy benchmarks.
arXiv:2606. 19319v1 Announce Type: cross Abstract: Production data integration is bottlenecked by repeated, lossy handoffs between data owners, engineers, and analysts who must collaboratively discover, structure, and query enterprise data.
arXiv:2607. 06229v1 Announce Type: cross Abstract: Major cloud data platforms now expose large language model capabilities as native SQL functions, enabling analysts to perform classification, filtering, sentiment analysis, extraction, similarity search, and aggregation within ordinary SQL queries.
arXiv:2606. 06462v1 Announce Type: new Abstract: Benchmarks are fundamental for evaluating and advancing LLMs and MLLMs by providing standardized and explicit measures of performance.
DI-Bench is a pipeline that automatically creates realistic data intelligence benchmarks for enterprise agents by linking data tables, dimensions, metrics, and documents into an artifact graph. It generates questions that combine structured data queries with knowledge retrieval, validates answers via query execution and LLM-generated questions, and has produced a 731-task benchmark covering knowledge retrieval, analytical computation, and rule‑grounded reasoning. Evaluation of four models on this benchmark shows that only 32% accuracy is achieved on computational tasks that involve business rules modifying the computation.
arXiv:2606. 12451v1 Announce Type: new Abstract: Large language models deployed as agents over large tool catalogs face a critical tool-retrieval bottleneck.