Smart Content Ingestion for Generative AI Workloads

The paper introduces a production-ready content‑extraction system tailored for generative AI workloads, addressing the heterogeneity of enterprise data formats such as PDFs, spreadsheets, and scanned documents. It features selective OCR routing, a scarcity‑first curation engine with a reference‑based extraction scorer, a deterministic structure‑aware chunker, and a read‑only retrieval evaluator that generates grounded questions and reports metrics like Hit@k and MRR. On a 180‑document corpus, the system achieves high accuracy (97.4/100 character score, 0.13% error rate) and strong retrieval performance (Hit@1 68.6%, Hit@10 92.8%, MRR 0.77).

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

• llms
• agents
• computer-vision