arXiv AI

SynthCorp: A Platform for Creating Benchmarks for Data Agents

SynthCorp is a platform that generates customizable data estates and benchmarks for large language model–driven data agents by simulating business events that create and modify corporate data over time. It constructs relational databases, document stores, and data catalogs, schedules event instances on a virtual calendar, executes them to produce a consistent data estate, and automatically generates analytical tasks with verified gold programs. The authors demonstrate SynthCorp by creating seven data estates with up to 1,000 operational tables and 256 tasks across five domains, achieving near‑zero error rates and enabling evaluation of models across different data contexts.

arXiv AI
Oct 5

Coco: An Agentic Copilot for the Hardware--Software Co-Design Lifecycle

arXiv:2610.02376v1 Announce Type: cross Abstract: Co-designing ML models and the accelerators that run them is an unusual reasoning task: architects must draw confident, high-stakes conclusions about...

By Samuel Kushnir, Kavya Sreedhar, Yeshwanth Reddy Pogula, Amir Yazdanbakhsh, Narges Shahidi, Ming Liu, Varun Gohil, Ravi Iyer, Parthasarathy Ranganathan, Christina Delimitrou, Suvinay Subramanian
arXiv AI
Oct 9

Synthesis Through Simulation: Generating Coherent Enterprise Data via Scalable Agent-System Interaction

The paper introduces Synthesis Through Simulation (STS), a schema‑free data synthesis method that uses a large language model agent to generate enterprise data by interacting with policy‑enforcing APIs in simulated environments. STS ensures structural validity by construction and separates validity enforcement from distribution modeling. The Generalist Populator, an agent within STS, achieves high distributional fidelity (0.88 average marginal fidelity) and full constraint satisfaction across ten environments without needing database schemas, outperforming traditional statistical synthesizers and schema‑privileged agents.

By Yipeng Li, Ashutosh Hathidara, Jane Lo, Harshavardhan Abichandani, Gunraj Singh, Atin Ghosh
arXiv AI
Oct 2

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

Argo-Bench is an evaluation framework that tests data agents on 210 enterprise-scale data science and analytics tasks, simulating a 2024 New York City food delivery platform with 81 million orders and 7.5 billion rows across 235 tables. Unlike traditional text‑to‑SQL benchmarks, the tasks require agents to navigate a realistic ERP warehouse, reconstruct facts, and perform actions such as banning accounts or allocating budgets, with outcomes scored by a simulator. Each task includes an executable reference solution that proves solvability using only the warehouse data.

By Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing, Duke Gand, Joseph J Ma
arXiv AI
Aug 12

DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?

arXiv:2608. 10366v1 Announce Type: new Abstract: Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments.

By Mizanur Rahman, Mohammed Saidul Islam, Ridwan Mahbub, Md Tahmid Rahman Laskar, Shafiq Joty, Enamul Hoque Prince
arXiv AI
Oct 15

A Governed Agent Architecture for Natural-Language Dashboard Generation over Enterprise DataBrain

The paper presents a DataBrain‑specific prototype that converts natural‑language analytic requests into inspectable dashboards. It combines a Codex plugin and an Agent Runtime that use schema‑constrained model decisions, typed tools, persistent state, and deterministic Hooks for approval, audit, checkpointing, recovery, and failure handling. The pipeline resolves entities, discovers metadata, enforces read‑only SQL, composes dashboards, and applies static checks, dynamic preflight, and browser inspection, with the model proposing actions while deterministic software controls execution and records state transitions.

By Yiou Wu, Zezhi Tang, Ningwei Bai, Liuhaichen Yang