The paper introduces SCSP, a training‑free framework that improves long‑context embeddings by selectively pooling informative tokens. SCSP partitions documents into sentence‑aware chunks, adds a semantic compression prompt to each chunk, and uses prompt‑isolated attention masks to estimate token importance. The selected tokens’ intermediate‑layer representations are aggregated to form the final embedding, yielding consistent performance gains across zero‑shot and fine‑tuned models on long‑context benchmarks.
arXiv:2606. 10716v1 Announce Type: cross Abstract: Pre-trained language models (PLMs) have achieved strong performance in keyphrase extraction (KPE), largely due to their ability to generate rich contextualized representations.
arXiv:2603. 05353v2 Announce Type: replace Abstract: Retrieval-augmented generation (RAG) for long-context question answering is bottlenecked by inference-time prefilling over large retrieved contexts.
SCOPE is a training‑free generative prompt‑compression framework that reduces LLM input length by chunking a prompt into semantically coherent segments, rewriting each chunk to be more concise, and then reconstructing a coherent prompt. Unlike token‑removal methods, SCOPE’s chunk‑level rewriting preserves critical information and text coherence, and includes optimization techniques for finer‑grained control of compression ratios. Extensive evaluations on question‑answering and summarization tasks show that SCOPE consistently outperforms selective compression baselines, especially at high compression ratios.
arXiv:2603. 06274v2 Announce Type: replace-cross Abstract: The quadratic computational complexity of self-attention remains a fundamental bottleneck for scaling Large Language Models (LLMs) to long contexts, particularly during the pre-filling phase.
The paper introduces K-Token Merging, a latent-space compression method that merges each contiguous block of K token embeddings into a single embedding using a lightweight encoder. The compressed sequence is then processed by a LoRA-adapted large language model, while generation continues in the original vocabulary. Experiments on tasks such as structural reasoning, sentiment classification, and code editing demonstrate that K-Token Merging achieves up to 75% input length reduction with minimal performance loss, placing it on the Pareto frontier of performance versus compression.
arXiv:2603. 01471v2 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.
Sentence-level AI-generated text detection (S-AGTD) for hybrid documents, where humans and LLMs co-author one text, faces two gaps: existing methods classify each sentence in isolation, discarding inter-sentence dependencies, and existing benchmarks omit the newest generation of generators. We construct MOSAIC, a benchmark of 16,000 hybrid documents over PubMed and XSum, generated by DeepSeek-V3.
arXiv:2607. 02980v1 Announce Type: cross Abstract: Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention.
Just Pass Twice (JPT) is a method that allows causal large language models to perform token classification for zero‑shot named entity recognition by concatenating the input with itself, giving each token full bidirectional context without architectural changes. The approach combines these representations with definition‑guided entity embeddings to enable flexible zero‑shot generalization. JPT achieves state‑of‑the‑art results, outperforming prior methods by an average of +7.9 F1 on CrossNER and MIT benchmarks and running over 20× faster than comparable generative approaches.
CausalEmbed is an auto‑regressive method for generating compact multi‑vector embeddings in visual document retrieval. By using iterative margin loss during contrastive training, it reduces the number of visual tokens needed by 30‑155× while keeping performance competitive across different backbones and benchmarks. The approach offers efficient training, scalable test‑time performance, and a flexible scaling strategy for multi‑vector representations.
arXiv:2606. 26875v1 Announce Type: cross Abstract: Reasoning capability has advanced rapidly in large language models (LLMs), leading to an increasing size of key-value (KV) cache in both prefilling and decoding stages.
arXiv:2603. 01471v3 Announce Type: replace-cross Abstract: Multimodal embedding models, rooted in multimodal large language models (MLLMs), have yielded significant performance improvements across diverse tasks such as retrieval and classification.