Institutional Books – Enriched Text is a 2025 release that transforms Harvard Library’s 983,004-volume collection (IB‑HL) into a multilingual, annotated dataset. The pipeline normalizes OCR text while preserving metadata, separating endmatter, detecting paragraph language, clustering duplicates, and scoring bits‑per‑byte, all wrapped in HTML‑like annotations. The resulting IB‑HL‑ET contains 217 B tokens across 983,003 volumes and 1.39 B annotated subtopic paragraphs, enabling users to customize output rather than accept a single editorial decision.
arXiv:2608. 08994v1 Announce Type: cross Abstract: Retrieving relevant evidence from noisy web data is challenging, particularly in sensitive domains containing incomplete reports, heterogeneous language, and irrelevant content.
By Joshua Castillo, Santosh Nukavarapu, Ravi Mukkamala
arXiv:2607. 22662v1 Announce Type: new Abstract: Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance.
By Peiguang Li, Yongwei Zhou, Juncheng Diao, Yuchun Fan, Jian Yang, Jianxiao Yang, Zhongda Su, Shuguang Jiao, Xiao Wei, Zhiye Zou, Gan Dong, Zhizhao Zeng, Rongxiang Weng, Jingang Wang, Xunliang Cai
Optimist Deep Research is an open‑source agent that conducts literature research as an iterative, state‑centric process. It breaks a research question into a tree of subquestions, uses search intents to query multiple scholarly databases, and merges results into a provenance‑preserving document registry. The agent iteratively expands the registry via citation snowballing, evaluates evidence, and refines the research plan through a critic that can add, split, rephrase, deprioritize, or close subquestions, ultimately producing a report with every citation linked to a retrieved document.
By Gihan Panapitiya
DeGenTWeb is a system that identifies websites whose informational prose is largely machine‑generated text (MGT) with minimal human input. By adapting MGT detectors for web pages and aggregating results across sites, the study reports that 20.9% of Common Crawl‑archived sites (Jan–Jul 2026) and 13.6% of Bing‑linked sites (2025) are MGT‑dominant, with a 0.12% mislabeling rate for human‑authored sites. The prevalence of MGT‑dominant sites is increasing over time and many share the same owner.
By Sichang Steven He, Calvin Ardi, Ramesh Govindan, Harsha V. Madhyastha
RIT‑RAG is a retrieval‑augmented generation framework that combines content retrieval with structural navigation of documents. It builds a tree for each document from its table of contents or sitemap, then uses retrieved chunk positions to induce manageable sub‑trees that an LLM agent can navigate, read selectively, and reformulate queries. Experiments on financial, scientific, and customer‑support benchmarks show RIT‑RAG outperforms vanilla, graph‑based, and agentic baselines, achieving up to 11.4‑point accuracy gains on the new EntQABench of 2.84 million technical‑documentation webpages.
By Meghanadh Pulivarthi, Swaraj Kumar Biswal, Kushagra Bhushan, Yatin Nandwani, Sachindra Joshi, Dinesh Raghu
arXiv:2607. 26070v1 Announce Type: cross Abstract: Large language model (LLM)-based agentic search systems are often evaluated as if the underlying LLM were the only component that matters, yet their measured performance also depends on the surrounding search environment: the Wikipedia snapshot, preprocessing pipeline, chunking policy, retrieval backend, tool schema, observation format, and answer submission rule.
By Guanming Xiong, Penghui Zhang
arXiv:2607. 10020v2 Announce Type: replace-cross Abstract: We present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus.
By Lars Henry Berge Olsen, Pierre Lison, Martin Jullum, Mark Anderson
arXiv:2609.05766v1 Announce Type: cross
Abstract: The dominant approach to building domain-specific pretraining corpora is to filter large web archives such as CommonCrawl. This works well for popula...
By Chirag Garg, Eelaaf Zahid, Farhan Ahmed, Jay Pankaj Gala, Eric Butler, Heiko Ludwig
arXiv:2607. 10020v1 Announce Type: cross Abstract: We present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus.
By Lars Henry Berge Olsen, Pierre Lison, Martin Jullum, Mark Anderson
arXiv:2609.37226v1 Announce Type: cross
Abstract: Answering questions and completing tasks over large document collections often requires connecting evidence spread across multiple documents, such as...
By Soyeong Jeong, Sujay Kumar Jauhar, Sung Ju Hwang, Andrew Joohun Nam
arXiv:2606.11350v2 Announce Type: replace
Abstract: Retrieval-augmented generation (RAG) degrades when applied to large, heterogeneous document collections. In RAG systems, dense similarity becomes l...
By Nabaraj Subedi, Ahmed Abdelaty, Shivanand Venkanna Sheshappanavar