← Back to all news
arXiv Machine Learning By Jacob Carlson

Interpretable Discovery from Unstructured Data: A High-Dimensional Approach

Read the original on arXiv Machine Learning →

The paper introduces a general-purpose framework that automatically discovers insights from unstructured data, such as text from open-ended surveys. It transforms these data into high-dimensional, structured datasets of interpretable concept measurements, defines concept-level parameters and null hypotheses, and applies validated high-dimensional multiple testing algorithms to select discoveries. The framework also generates and evaluates human-interpretable natural language descriptions of the findings, enabling fast, inexpensive sensitivity analysis and replication.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 25

The Measurement Revolution? Credible Measurement and Inference in the Age of AI

The article discusses how artificial intelligence is reshaping measurement in economics by converting unstructured data into structured variables at low cost, enabling large‑scale measurement that was previously infeasible. It outlines three stages—discovery, construct definition, and observation—where AI impacts the measurement pipeline and stresses the importance of rigorous validation to ensure credible inference. The review offers guidance on navigating the shift from a single scalable measure to multiple plausible ones that can lead to differing empirical conclusions.

By Melissa Dell, Ashesh Rambachan
More like this →
arXiv Machine Learning
Sep 11

AI Economist Agent: An Agentic Framework for Evidence-Based Economic and Financial Analysis with RAG, Knowledge Graphs, and Large Language Models

The paper introduces an AI economist agent that integrates large language models, retrieval‑augmented generation, knowledge graphs, and quantitative models to conduct evidence‑based economic and financial scenario analysis. The framework orchestrates LLM agents to plan analyses, retrieve relevant evidence, and structure economic mechanisms, while registered quantitative models produce numerical outcomes and predefined tests validate intermediate results for inclusion in the final report. Applied to European macro‑financial stress scenarios and bank capital analysis, the empirical study demonstrates the agent’s ability to combine flexible evidence retrieval and scenario construction while maintaining traceability to sources and explicit model calculations.

By Masahiro Kato
llmsragagents
More like this →
arXiv AI
Jun 2

AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science

arXiv:2603. 19005v2 Announce Type: replace-cross Abstract: Data science plays a critical role in transforming complex data into actionable insights across numerous domains.

By An Luo, Jin Du, Xun Xian, Robert Specht, Fangqiao Tian, Ganghua Wang, Xuan Bi, Charles Fleming, Ashish Kundu, Jayanth Srinivasa, Mingyi Hong, Rui Zhang, Tianxi Li, Galin Jones, Jie Ding
llmsagentsbenchmarks
More like this →