The Era by Eon Benchmark is a new dataset for evaluating large language model agents that interact with enterprise tools. It constructs a complete fictional company with product simulators, internal databases, and benchmark questions, all generated from a shared entity graph to ensure consistency. Exact answer keys are computed from the generated records, allowing precise grading and validation of realism and adversarial robustness across 23 simulated companies.
By Benjamin Gruenbaum, Doron Porat, Assaf Natanzon, Roy Zavida, Chen Dinachi, Or Itzahary
Argo-Bench is an evaluation framework that tests data agents on 210 enterprise-scale data science and analytics tasks, simulating a 2024 New York City food delivery platform with 81 million orders and 7.5 billion rows across 235 tables. Unlike traditional text‑to‑SQL benchmarks, the tasks require agents to navigate a realistic ERP warehouse, reconstruct facts, and perform actions such as banning accounts or allocating budgets, with outcomes scored by a simulator. Each task includes an executable reference solution that proves solvability using only the warehouse data.
By Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing, Duke Gand, Joseph J Ma
The paper introduces inexpensive, scalable methods for evaluating language model behavior across different vendors and releases. By running identical public stimuli on a cross‑vendor panel and analyzing transcripts via exact match, LLM‑coded codebooks, or instrumented environments, the authors can quantify model responses at a cost of a few dollars per model. Applying these tools to four years of releases reveals patterns of convergence, resistance, positional stability, and compliance that vary by generation, lab, and harness.
By Tapan Parikh
arXiv:2609.16487v1 Announce Type: new
Abstract: We present a framework for evaluating data-science agents on live, continuously updated data using executable ground truth and format-agnostic factoid...
By Aniruddha Tamhane, Raghavendra Addanki, Ayushi Aggarwal, Aditya Bansal, Rui Wang, Charles Menguy, Swati Jain
SynthCorp is a platform that generates customizable data estates and benchmarks for large language model–driven data agents by simulating business events that create and modify corporate data over time. It constructs relational databases, document stores, and data catalogs, schedules event instances on a virtual calendar, executes them to produce a consistent data estate, and automatically generates analytical tasks with verified gold programs. The authors demonstrate SynthCorp by creating seven data estates with up to 1,000 operational tables and 256 tasks across five domains, achieving near‑zero error rates and enabling evaluation of models across different data contexts.
By Jiani Zhang, Manav Garg, Eugenie Lai, Cheng Zhen, Eva Sharma, Yang Pan, Yaming Lin, Alon Halevy
The paper introduces GROUND, a framework that limits large language model (LLM) analytics to a governed semantic layer for enterprise data warehouses. GROUND supplies approved metrics, dimensions, join paths, filters, and security rules, then validates generated SQL against these constraints before execution, retrying or abstaining on violations. In benchmarks, GROUND eliminates hallucinations across all evaluated categories and prevents row‑level security breaches, outperforming schema‑only, schema‑RAG, and semantic‑only approaches.
By Aravind Sasidharan Pillai