Towards Data Science

One Document Type, a Million Files: Structured Extraction into the SQL Table RAG Queries

The article discusses a case study in enterprise document intelligence where a single document type contains a million files. It outlines a workflow that takes about an hour with two people to extract six to ten structured fields, emphasizing the importance of identifying two key signals that distinguish a valid column from one that could break a filter later. The focus is on converting unstructured documents into a structured SQL table for Retrieval-Augmented Generation (RAG) queries.

Towards Data Science
Aug 22

Multi-Document RAG: A Folder of Unrelated PDFs Is One Long Document with a Nested Outline

The article discusses a method for handling a folder of unrelated PDFs as a single long document with a nested outline. It highlights that without shared fields, an index cannot be built, so the approach uses one summary line per file and each file’s own table of contents, with retrieval routes extending down two levels. This structure enables retrieval-augmented generation (RAG) across multiple documents.

By angela shi
Towards Data Science
Sep 29

I Compacted 1,000 Apache Iceberg Files Into 6. Here’s What Happened to Query Performance.

The article reports on a benchmark that reduced 1,000 Apache Iceberg files to just six, then measured how this consolidation affected query performance across three SQL workloads. It details the methodology and results of the experiment, highlighting changes in execution speed and resource usage. The findings illustrate the trade‑offs between file count and query efficiency in large data systems.

By Thomas Reid
Towards Data Science
Aug 23

Parse the Folder, Not Just the PDFs: The Relational Tables RAG Needs on a Case File

The article discusses how Enterprise Document Intelligence should begin by parsing the folder structure rather than just PDFs, emphasizing that the index must reflect the case type’s requirements before any folder is accessed. It highlights that the two key questions to develop are not retrieval questions but rather focus on the relational tables needed for Retrieval-Augmented Generation (RAG) in a case file context.

By angela shi