Hugging Face Trending Papers

Institutional Books - Enriched Text: A customizable multilingual open-source pipeline for denoising, deduplicating, and annotating OCR text at scale

Read the original on Hugging Face Trending Papers →

Institutional Books – Enriched Text is a 2025 release that transforms Harvard Library’s 983,004-volume collection (IB‑HL) into a multilingual, annotated dataset. The pipeline normalizes OCR text while preserving metadata, separating endmatter, detecting paragraph language, clustering duplicates, and scoring bits‑per‑byte, all wrapped in HTML‑like annotations. The resulting IB‑HL‑ET contains 217 B tokens across 983,003 volumes and 1.39 B annotated subtopic paragraphs, enabling users to customize output rather than accept a single editorial decision.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at Hugging Face Trending Papers.

arXiv Computation and Language
Oct 27

RefinedGAW: A Refined German Academic Web Dataset

arXiv:2610.34572v1 Announce Type: new Abstract: The largest open German text datasets are built from broad web crawls, which provide scale but limited coverage of specific domains. We introduce Refin...

By Maximilian Idahl, Oleh Astappiev, Benedikt Droste, Robert Scholz, Bj\"orn Pl\"uster, Jan Philipp Harries, Robert J\"aschke, Wolfgang Nejdl
arXiv AI
Jul 28

CuraWeb: Joint Optimization of Quality, Redundancy, and Diversity for Web-Scale Pretraining Data

arXiv:2607. 22662v1 Announce Type: new Abstract: Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance.

By Peiguang Li, Yongwei Zhou, Juncheng Diao, Yuchun Fan, Jian Yang, Jianxiao Yang, Zhongda Su, Shuguang Jiao, Xiao Wei, Zhiye Zou, Gan Dong, Zhizhao Zeng, Rongxiang Weng, Jingang Wang, Xunliang Cai
arXiv AI
Oct 9

RIT-RAG: Navigating Document Corpora with Retrieval-Induced Trees

RIT‑RAG is a retrieval‑augmented generation framework that combines content retrieval with structural navigation of documents. It builds a tree for each document from its table of contents or sitemap, then uses retrieved chunk positions to induce manageable sub‑trees that an LLM agent can navigate, read selectively, and reformulate queries. Experiments on financial, scientific, and customer‑support benchmarks show RIT‑RAG outperforms vanilla, graph‑based, and agentic baselines, achieving up to 11.4‑point accuracy gains on the new EntQABench of 2.84 million technical‑documentation webpages.

By Meghanadh Pulivarthi, Swaraj Kumar Biswal, Kushagra Bhushan, Yatin Nandwani, Sachindra Joshi, Dinesh Raghu