arXiv:2607. 01601v1 Announce Type: new Abstract: Large scale document deduplication must preserve semantic equivalence while remaining efficient over massive corpora.
Large scale document deduplication must preserve semantic equivalence while remaining efficient over massive corpora. We present SemHash LLM, a multi granularity framework that unifies semantic projection hashing, attention weighted MinHash, contrastive boundary learning, and selective LLM based adjudication.
Large-scale pretraining corpora contain substantial duplicate content. Although document-level deduplication is widely used, removing subdocument-level redundancy remains challenging.
arXiv:2606. 28057v1 Announce Type: cross Abstract: Language models (LMs) represent tokens using embedding matrices that scale linearly with the vocabulary size.
arXiv:2607. 22662v1 Announce Type: new Abstract: Open-web corpora curated via highly selective filters, such as FineWeb-Edu and DCLM, constitute the core of LLM pretraining data and have significantly advanced LLM performance.
arXiv:2607. 24332v1 Announce Type: cross Abstract: Common chunking strategies in Retrieval-Augmented Generation (RAG) systems often create redundant chunks.
arXiv:2607. 10020v2 Announce Type: replace-cross Abstract: We present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus.
arXiv:2608. 03859v1 Announce Type: cross Abstract: Large language models (LLMs) pose challenges to academic integrity and peer review.
arXiv:2607. 01445v1 Announce Type: cross Abstract: Malware poses a critical and ever-evolving threat, and robust and effective systems for detecting and classifying malware are of essential importance.
arXiv:2607. 10020v1 Announce Type: cross Abstract: We present FindMyText, an open-source Python package designed to efficiently assess whether a given text appears, in part or in full, within a text corpus.
arXiv:2606. 20089v1 Announce Type: cross Abstract: Persian pretrained language models (PLMs) are still limited by the scarcity of large-scale, high-quality pretraining corpora and by insufficient evaluation beyond standard classification and NER tasks.
Malware poses a critical and ever-evolving threat, and robust and effective systems for detecting and classifying malware are of essential importance. $n$-grams features are among the common static features used in effective machine learning systems for malware, but these features are inherently brittle.
arXiv:2512. 10388v3 Announce Type: replace-cross Abstract: Conventional Sequential Recommender Systems (SRS) typically assign unique hash IDs (HID) to construct item embeddings, which mainly capture collaborative signals from historical user-item interactions.