arXiv:2607. 10020v1 Announce Type: cross Abstract: We present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus.
By Lars Henry Berge Olsen, Pierre Lison, Martin Jullum, Mark Anderson
arXiv:2502.10673v2 Announce Type: replace-cross
Abstract: Retrieval-Augmented Generation (RAG) has become an effective method for enhancing large language models (LLMs) with up-to-date knowledge. How...
By Yepeng Liu, Xuandong Zhao, Dawn Song, Yuheng Bu
Institutional Books – Enriched Text is a 2025 release that transforms Harvard Library’s 983,004-volume collection (IB‑HL) into a multilingual, annotated dataset. The pipeline normalizes OCR text while preserving metadata, separating endmatter, detecting paragraph language, clustering duplicates, and scoring bits‑per‑byte, all wrapped in HTML‑like annotations. The resulting IB‑HL‑ET contains 217 B tokens across 983,003 volumes and 1.39 B annotated subtopic paragraphs, enabling users to customize output rather than accept a single editorial decision.
Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level redundancy remains challenging.
arXiv:2602. 10908v2 Announce Type: replace-cross Abstract: We present SoftMatcha 2, an ultra-fast and flexible search algorithm that enables search over trillion-scale natural language corpora in under 0.
By Masataka Yoneda, Yusuke Matsushita, Go Kamoda, Kohei Suenaga, Takuya Akiba, Masaki Waga, Sho Yokoi
arXiv:2605. 28510v2 Announce Type: replace-cross Abstract: Large language models (LLMs) for code completion and generation are increasingly used in software development, yet they may reproduce training examples verbatim and without authorship attribution, raising legal and ethical concerns around plagiarism and license compliance.
By Andrea Gurioli, Davide D'Ascenzo, Federico Pennino, Maurizio Gabbrielli, Stefano Zacchiroli