arXiv AI By Jiani Zhang, Manav Garg, Eugenie Lai, Cheng Zhen, Eva Sharma, Yang Pan, Yaming Lin, Alon Halevy

SynthCorp: A Platform for Creating Benchmarks for Data Agents

Read the original on arXiv AI →

SynthCorp is a platform that generates customizable data estates and benchmarks for large language model–driven data agents by simulating business events that create and modify corporate data over time. It constructs relational databases, document stores, and data catalogs, schedules event instances on a virtual calendar, executes them to produce a consistent data estate, and automatically generates analytical tasks with verified gold programs. The authors demonstrate SynthCorp by creating seven data estates with up to 1,000 operational tables and 256 tasks across five domains, achieving near‑zero error rates and enabling evaluation of models across different data contexts.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 5

Coco: An Agentic Copilot for the Hardware--Software Co-Design Lifecycle

arXiv:2610.02376v1 Announce Type: cross Abstract: Co-designing ML models and the accelerators that run them is an unusual reasoning task: architects must draw confident, high-stakes conclusions about...

By Samuel Kushnir, Kavya Sreedhar, Yeshwanth Reddy Pogula, Amir Yazdanbakhsh, Narges Shahidi, Ming Liu, Varun Gohil, Ravi Iyer, Parthasarathy Ranganathan, Christina Delimitrou, Suvinay Subramanian
arXiv AI
Oct 9

Synthesis Through Simulation: Generating Coherent Enterprise Data via Scalable Agent-System Interaction

The paper introduces Synthesis Through Simulation (STS), a schema‑free data synthesis method that uses a large language model agent to generate enterprise data by interacting with policy‑enforcing APIs in simulated environments. STS ensures structural validity by construction and separates validity enforcement from distribution modeling. The Generalist Populator, an agent within STS, achieves high distributional fidelity (0.88 average marginal fidelity) and full constraint satisfaction across ten environments without needing database schemas, outperforming traditional statistical synthesizers and schema‑privileged agents.

By Yipeng Li, Ashutosh Hathidara, Jane Lo, Harshavardhan Abichandani, Gunraj Singh, Atin Ghosh
arXiv AI
Oct 2

Argo-Bench: Evaluating Data Agents on Enterprise-Scale Workflows

Argo-Bench is an evaluation framework that tests data agents on 210 enterprise-scale data science and analytics tasks, simulating a 2024 New York City food delivery platform with 81 million orders and 7.5 billion rows across 235 tables. Unlike traditional text‑to‑SQL benchmarks, the tasks require agents to navigate a realistic ERP warehouse, reconstruct facts, and perform actions such as banning accounts or allocating budgets, with outcomes scored by a simulator. Each task includes an executable reference solution that proves solvability using only the warehouse data.

By Gabriel Tomitsuka, Arman Raayatsanati, Emma Xing, Duke Gand, Joseph J Ma