arXiv AI

Generating a Consistent Enterprise: Synthesis and Reference-Free Evaluation of Multi-System Business Data

The paper introduces a synthetic data generator that creates fully consistent, fictional enterprises—complete with workforce, customers, sales, support, and communication records—without relying on any real dataset. It validates realism through a five‑axis scorecard, an adversarial detector, and soundness checks, achieving a mean realism score of 99.1 across 23 generated companies. A second generator produces relational databases from business questions, ensuring qualifying rows and exact labels, and is available as a hosted service and containerized simulators.

arXiv AI
Sep 11

The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents

The Era by Eon Benchmark is a new dataset for evaluating large language model agents that interact with enterprise tools. It constructs a complete fictional company with product simulators, internal databases, and benchmark questions, all generated from a shared entity graph to ensure consistency. Exact answer keys are computed from the generated records, allowing precise grading and validation of realism and adversarial robustness across 23 simulated companies.

By Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
arXiv AI
Oct 2

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

Argo-Bench is an evaluation framework that tests data agents on 210 enterprise-scale data science and analytics tasks, simulating a 2024 New York City food delivery platform with 81 million orders and 7.5 billion rows across 235 tables. Unlike traditional text‑to‑SQL benchmarks, the tasks require agents to navigate a realistic ERP warehouse, reconstruct facts, and perform actions such as banning accounts or allocating budgets, with outcomes scored by a simulator. Each task includes an executable reference solution that proves solvability using only the warehouse data.

By Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing, Duke Gand, Joseph J Ma
arXiv AI
Sep 25

Low-Cost Assays for Measuring Model Behavior Across Vendors and Releases

The paper introduces inexpensive, scalable methods for evaluating language model behavior across different vendors and releases. By running identical public stimuli on a cross‑vendor panel and analyzing transcripts via exact match, LLM‑coded codebooks, or instrumented environments, the authors can quantify model responses at a cost of a few dollars per model. Applying these tools to four years of releases reveals patterns of convergence, resistance, positional stability, and compliance that vary by generation, lab, and harness.

By Tapan Parikh
arXiv AI
3d ago

SynthCorp: A Platform for Creating Benchmarks for Data Agents

SynthCorp is a platform that generates customizable data estates and benchmarks for large language model–driven data agents by simulating business events that create and modify corporate data over time. It constructs relational databases, document stores, and data catalogs, schedules event instances on a virtual calendar, executes them to produce a consistent data estate, and automatically generates analytical tasks with verified gold programs. The authors demonstrate SynthCorp by creating seven data estates with up to 1,000 operational tables and 256 tasks across five domains, achieving near‑zero error rates and enabling evaluation of models across different data contexts.

By Jiani Zhang, Manav Garg, Eugenie Lai, Cheng Zhen, Eva Sharma, Yang Pan, Yaming Lin, Alon Halevy
arXiv AI
Aug 28

GROUND: Reducing Hallucinations in LLM-Based Enterprise Analytics Through Governed Semantic Definitions

The paper introduces GROUND, a framework that limits large language model (LLM) analytics to a governed semantic layer for enterprise data warehouses. GROUND supplies approved metrics, dimensions, join paths, filters, and security rules, then validates generated SQL against these constraints before execution, retrying or abstaining on violations. In benchmarks, GROUND eliminates hallucinations across all evaluated categories and prevents row‑level security breaches, outperforming schema‑only, schema‑RAG, and semantic‑only approaches.

By Aravind Sasidharan Pillai
arXiv AI
Sep 28

CARGO: Context-Aware Retrieval-Gated Evaluation of Agentic AI in Production

CARGO is a framework for evaluating agentic AI systems in production that addresses the problem of reference-instance divergence (RID), where reference-based judges penalize correct answers that involve different entity identifiers. It treats retrieved references as procedural exemplars, grounds judgments in the live instance’s context, assigns a three-way status to claims, and gates evaluation by retrieval confidence. Using the CARGO-Bench diagnostic suite, CARGO eliminates false penalties and improves discrimination while revealing a limitation in detecting procedural corruptions.

By Mukul Chhabra, Shail Patel, Luigi Medrano