arXiv Machine Learning

Interpretable Discovery from Unstructured Data: A High-Dimensional Approach

The paper introduces a general-purpose framework that automatically discovers insights from unstructured data, such as text from open-ended surveys. It transforms these data into high-dimensional, structured datasets of interpretable concept measurements, defines concept-level parameters and null hypotheses, and applies validated high-dimensional multiple testing algorithms to select discoveries. The framework also generates and evaluates human-interpretable natural language descriptions of the findings, enabling fast, inexpensive sensitivity analysis and replication.

arXiv AI
Aug 25

The Measurement Revolution? Credible Measurement and Inference in the Age of AI

The article discusses how artificial intelligence is reshaping measurement in economics by converting unstructured data into structured variables at low cost, enabling large‑scale measurement that was previously infeasible. It outlines three stages—discovery, construct definition, and observation—where AI impacts the measurement pipeline and stresses the importance of rigorous validation to ensure credible inference. The review offers guidance on navigating the shift from a single scalable measure to multiple plausible ones that can lead to differing empirical conclusions.

By Melissa Dell, Ashesh Rambachan
arXiv Machine Learning
Sep 11

AI Economist Agent: An Agentic Framework for Evidence-Based Economic and Financial Analysis with RAG, Knowledge Graphs, and Large Language Models

The paper introduces an AI economist agent that integrates large language models, retrieval‑augmented generation, knowledge graphs, and quantitative models to conduct evidence‑based economic and financial scenario analysis. The framework orchestrates LLM agents to plan analyses, retrieve relevant evidence, and structure economic mechanisms, while registered quantitative models produce numerical outcomes and predefined tests validate intermediate results for inclusion in the final report. Applied to European macro‑financial stress scenarios and bank capital analysis, the empirical study demonstrates the agent’s ability to combine flexible evidence retrieval and scenario construction while maintaining traceability to sources and explicit model calculations.

By Masahiro Kato
arXiv AI
Jun 2

AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science

arXiv:2603. 19005v2 Announce Type: replace-cross Abstract: Data science plays a critical role in transforming complex data into actionable insights across numerous domains.

By An Luo, Jin Du, Xun Xian, Robert Specht, Fangqiao Tian, Ganghua Wang, Xuan Bi, Charles Fleming, Ashish Kundu, Jayanth Srinivasa, Mingyi Hong, Rui Zhang, Tianxi Li, Galin Jones, Jie Ding
arXiv AI
Aug 28

Nomad: Autonomous Exploration and Discovery

Nomad is an autonomous system designed to explore and discover insights within large data corpora. It builds an explicit Exploration Map to systematically traverse a domain, generating and testing hypotheses with an explorer agent that leverages document, web, and database searches. After verification, it produces cited reports and meta-reports, and its evaluation framework assesses trustworthiness, quality, and diversity, showing superior performance over baselines on UN, WHO, and arXiv datasets.

By Bokang Jia, Samta Kamboj, Satheesh Katipomu, Seung Hun Han, Neha Sengupta, Andrew Jackson