Retrieval-augmented generation

Retrieval pipelines, vector search, chunking and reranking: how models are grounded in a corpus instead of their weights.

6,126 stories · RSS feed

arXiv Machine Learning
Oct 12

Differentiable Retrieval-Augmented Generation for Predicting Cellular Responses to Gene Perturbation

The paper introduces PT‑RAG, a retrieval‑augmented generation framework that enhances cellular perturbation response prediction by incorporating context from related perturbations. PT‑RAG uses a two‑stage retrieval process—semantic retrieval to shortlist candidate perturbations followed by a differentiable Gumbel‑Softmax selector that adapts the selection based on cell state and query perturbation. Experiments on two backbones, a frozen STATE‑style generator and a fully trained scGPT, show that PT‑RAG improves distributional similarity and predictive quality, achieving a 5.9% reduction in 2‑Wasserstein distance on cross‑cell‑type tasks.

By Andrea Giuseppe Di Francesco, Andrea Rubbi, Rishabh Jain, Pietro Li\`o
arXiv Machine Learning
Oct 12

Linearized Wasserstein dimensionality reduction with approximation guarantees

The paper introduces LOT Wassmap, an algorithm designed to uncover low‑dimensional structures in the Wasserstein space without computing a full pairwise distance matrix. It leverages approximation schemes such as Sinkhorn distances and linearized optimal transport to accelerate computations, and provides theoretical guarantees on embedding quality even when only finite samples are available. Experiments show that LOT Wassmap achieves accurate embeddings that improve with larger sample sizes while significantly reducing computational cost compared to traditional pairwise‑distance‑based methods.

By Alexander Cloninger, Keaton Hamm, Varun Khurana, Caroline Moosm\"uller
arXiv Computer Vision
Oct 12

Shift-Aware Identity-Guided Latent Refinement for Referring Audio--Visual Segmentation

The paper introduces Shift-Aware Identity-guided Latent (SAIL) refinement for Referring Audio‑Visual Segmentation (Ref‑AVS), a task that requires precise segmentation of a specific object using both visual and auditory cues grounded in natural language. SAIL employs a dual‑token strategy to separate frame‑level dynamics from a global semantic anchor, a shift‑aware refinement module to correct target‑shift drift, and a decoder with continuous memory updates to preserve spatio‑temporal coherence and identity consistency. Experiments on two Ref‑AVS benchmarks show that SAIL outperforms existing AVS, Ref‑VOS, and Ref‑AVS methods.

By Kun Li, Sami Sebastian Brandt, Michael Ying Yang
arXiv Computer Vision
Oct 12

Automated Detection and Encoding of Cartographic Signs in Historical Maps

The paper introduces a two-stage framework for detecting cartographic signs in historical maps and generating interpretable embeddings that capture both semantics and stylistic variations. It presents CaSiDD, an open benchmark dataset covering multiple sign categories and diverse cartographic styles, and demonstrates the effectiveness of YOLOv10-X for automated sign detection. The authors also develop a method based on an adapted RADIOv4 backbone to encode sign form and semantics while suppressing confounding factors, providing a foundation for scalable evidence extraction in cartographic culture and geohistorical research.

By Remi Petitpierre
arXiv Computer Vision
Oct 12

Beyond a Single Crop: Contextualized Embeddings for Player Re-Identification

The paper introduces ContextReID, a framework that generates player embeddings by incorporating contextual information from co-occurring players in the same frame and from multiple frames in a broadcast sequence. Unlike traditional single-crop methods, this approach uses a vision‑language model to produce context‑aware embeddings that capture cross‑frame identity relationships. Experiments on three multi‑object tracking benchmarks demonstrate that ContextReID outperforms conventional single‑crop embeddings in Rank‑1 and mAP metrics for player re‑identification.

By Ge Luo, Changhuo Yang, Weisi Fan, Rui Yang, Shaokai Wang
arXiv Computer Vision
Oct 12

CoVA: Composing VAE Representations for Generative Learning

CoVA introduces a lightweight framework that composes VAE features online to provide stage‑wise guidance for flow‑matching training, improving semantic discrimination and spatial structure without extra inference overhead. By jointly training a small latent composer with class supervision, masked reconstruction, and flip consistency, CoVA aligns intermediate SiT features early and later uses detached composer posteriors for sample‑dependent soft‑label conditioning. On ImageNet 256×256, CoVA cuts SiT‑XL/2 FID from 17.2 to 7.4 in 400K steps and achieves an FID of 2.03 in only 400 epochs with classifier‑free guidance, adding just 8.5 M training‑only parameters.

By Zhaowei Chen, Yikai Ge, Linze Li, Yao Tang, Xiang Li
arXiv Computer Vision
Oct 12

T2M Mamba: Motion Periodicity-Saliency Coupling Approach for Stable Text-Driven Motion Generation

The paper introduces T2M Mamba, a text‑to‑motion generation model that addresses two key shortcomings of prior work: the independent treatment of motion periodicity and keyframe saliency, and sensitivity to paraphrased text. It achieves this by employing a Periodicity‑Saliency Aware Mamba module that estimates keyframe weights with Density Peaks Clustering and motion periodicity via FFT‑accelerated autocorrelation, and a Periodic Differential Cross‑modal Alignment Module (PDCAM) that improves alignment between textual and motion embeddings. Experiments on HumanML3D and KIT‑ML datasets demonstrate significant performance gains, achieving an FID of 0.068 and improvements across all other metrics.

By Xingzu Zhan, Chen Xie, Honghang Chen, Yixun Lin, Xiaochun Mai
arXiv AI
Oct 12

SmartSalesBench: A Resistance-Aware Multi-Turn Benchmark for Evaluating LLM Sales Agents

SmartSalesBench is a new multi‑turn conversational benchmark designed to evaluate large language model sales agents against buyer personas that are explicitly programmed to resist and challenge the agents. The benchmark features five agent profiles, twelve resistant buyer personas, fifteen smartphone products across six price tiers, and includes 13,680 sessions across four controlled experiments. Evaluation uses nine metrics grouped into primary, secondary, and exploratory tiers, combining automated LLM‑as‑judge scoring with human assessment.

By P N Bhargav Teja, Velpula Lokesh Reddy, Madhu Basumata, Susen Sarkar, Rajasekhara Reddy Duvvuru Muni
arXiv AI
Oct 12

Locate, Judge, Read: Role-Specialized Tool Agents over a Knowledge-Point Index for Open-Domain Multimodal Document QA

KnowForge is a modular pipeline for open‑domain multimodal document question answering that separates retrieval, judging, and reading into distinct agents. It uses a hybrid‑retrieval knowledge‑point index, a Locator agent for searching, a Bridge Judge to enforce a second hop, a Deep Reader that handles text, images, and tables, and a synthesizer with format checking. On the M3DocVQA benchmark it achieves an F1 of 47.5, outperforming prior systems by 11 points, and shows strong gains on MultiHop‑RAG and MMLongBench‑Doc as well.

By Wenlin Lin, Fanzhe Wei
arXiv AI
Oct 12

Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis

PRISM (Population‑Retrieved Inference for Structured Multimodal Prognosis) is a multimodal language model that integrates structured measurements, physiological waveforms, longitudinal history, and population context for clinical prognosis. It uses per‑feature numerical tokenization, recency‑weighted aggregation of ECGs, and a dual‑modality population memory bank, retrieving mean‑pooled embeddings of similar encounters without including individual records in the prompt. On the 1,443‑task MDS‑ED benchmark, PRISM achieves high AUROCs across diagnosis, deterioration, ICU admission, and mortality, outperforming baselines and demonstrating the value of structured multimodal integration and supervised external adaptation.

By Hamza Shafiq, Jialu Tang, Ting Dang, Jun Hu, Aaqib Saeed
arXiv AI
Oct 12

LeWAM: End-to-End World and Action Modeling with JEPAs

LeWAM is an end‑to‑end joint‑embedding predictive architecture that learns compact latent world models without pixel reconstruction and directly supports policy execution and planning. It trains jointly on next‑latent‑state prediction and action‑flow matching while imposing an isotropic Gaussian constraint on the latent state. The approach yields significant performance gains, raising visual‑planning success rates from 28.6 % to 89.7 % on contact‑rich tasks, from 10.9 % to 46.2 % on long‑horizon tasks, and speeding up planning by 32.3×, while remaining competitive on behavior‑cloning tasks and providing interpretable control‑relevant representations.

By Minghao Fu, Tavis Siebert, Eryk Halicki, Randall Balestriero
arXiv AI
Oct 12

Global Consistency Repair for Natural-Language Claim Sets

The paper introduces QXR, an adaptive framework that repairs inconsistent natural‑language claim sets by treating a large language model as a noisy consistency judge. QXR identifies minimal conflicting subsets and removes a small hitting set of claims, thereby preserving the largest coherent subset of the original set. Experiments on benchmarks derived from VitaminC, FEVER, and conflicting‑evidence RAG show that QXR retains substantially more valid claims than all‑at‑once judging while maintaining high precision.

By Paul He, Elke Kirschbaum, Shiva Kasiviswanathan
arXiv AI
Oct 12

First-Order In-Context Shapley: Scalable Data Auditing for LLM Alignment and Evaluation

The paper introduces First-Order In-Context Shapley, an inference‑only method that evaluates data utility for large language model alignment by computing the first‑order term of an in‑context cooperative game over semantic k‑NN neighborhoods in O(k) forward passes. It projects log‑likelihood shifts onto a directed influence graph, yielding three signals—Exemplifying Advantage, Exemplified Advantage, and One‑of‑a‑Kind Score—to identify high‑utility demonstrations and rare prompts. The approach reduces compute by 19× compared to Monte‑Carlo Shapley, filters 1–5% of records, and improves error detection and reward accuracy in DPO training and benchmark auditing. whyItMatters":"By enabling scalable, inference‑only data auditing, the method addresses label noise and safety inversions that hinder post‑training alignment of large language models, leading to measurable improvements in model reliability and reward accuracy."

By Yunting Song, Matthew Watson, Moustafa Farid Alzantot, Peter Grabowski, Jun Qin
arXiv Computation and Language
Oct 12

GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

GeopoliticalQA is a new benchmark for retrieval‑augmented question answering over 415 open‑source geopolitical and strategic intelligence reports, containing 5,525 document‑grounded question‑answer pairs that cover factual, inferential, comparative, and risk‑oriented queries. The authors propose a document‑aware generation pipeline that profiles reports before generating questions, and they validate dataset quality through expert annotation and an LLM‑as‑judge protocol. Experiments on the benchmark show that while retrieval performance is similar across models at the document level, chunk‑level retrieval is not driven by embedding‑model scale, and answer generation remains difficult even with gold evidence, especially for inferential and risk‑oriented questions.

By Veronika Solopova, Ivan Havlytskyi, Viktoria Skorik, Ostap Vykhopen
arXiv Computation and Language
Oct 12

CliQ-RRG: Clinical-Knowledge Guided Disease-aware Visual-Textual Alignment for QA-Style Radiology Report Generation

CliQ‑RRG is a two‑stage framework for generating radiology reports that aligns visual and textual data using disease‑aware embeddings and prior‑guided attention to capture multi‑view dependencies. The second stage retrieves graph‑grounded clinical evidence and employs a large language model to produce concise, interpretable question‑answer style reports with diagnostic summaries. Experiments on two chest X‑ray benchmarks show that CliQ‑RRG outperforms previous methods on both lexical and clinical metrics, producing accurate and clinically coherent reports.

By Chandravardhan Singh Raghaw, Shahid Shafi Dar, Arnav Nishith Sharma, Shrey Modi, Nagendra Kumar
arXiv Computation and Language
Oct 12

Vision Transformers Learn Word Meaning from Images Alone

The study demonstrates that large DINOv3 vision transformers, trained solely on images without any language supervision, can organize rendered words in their embedding space according to meaning. By deriving word representations from differences between word images and matched controls, the authors show that semantically related words cluster together, brand names group by industry, and country-capital pairs exhibit consistent vector offsets. These representations align with visual object categories and share structure with text and speech encoders, indicating that image-only training can impart both visual and semantic understanding of words.

By Anders Gj{\o}lbye, Sebastian Ray Mason, Stella Frank, Lars Kai Hansen
arXiv Computation and Language
Oct 12

Beyond the Final Embedding: Retrieving Memories from Token-Level Evidence

The paper introduces a memory-augmented retrieval method that leverages token-level evidence instead of relying solely on single-vector embeddings. By performing sparse token matching and aggregating scores with whole-memory similarity, the approach improves retrieval performance across eight benchmarks and various model families without extra training. An optional low-rank query adapter can further boost results when supervised.

By Han Luo, Zhan Zhuang, Xu He, Jujia Zhao, Yu Li, Suqin Yuan, Dong Li, Lei Feng
arXiv Computation and Language
Oct 12

Lychee-Reranker: Fast but Not Late Interaction for Compressed Document Reranking

Lychee‑Reranker is a fast, not‑late‑interaction reranker that decouples query and passage computation while preserving expressive relevance modeling. It uses an encoder–decoder architecture: passages are pre‑encoded with Matryoshka embedding pooling, and the decoder jointly processes system/user instructions and query intent, with cross‑attention capturing fine‑grained relevance. The design yields four benefits—offline passage encoding for efficiency, cross‑attention expressiveness, compact Matryoshka pooling, and adjustable test‑time compute—and is available in Nano, Small, and Large sizes (0.27B, 1B, and 4B activated parameters).

By Xinping Zhao, Jiaxin Xu, Ziqi Dai, Xin Zhang, Huiyao Chen, Shouzheng Huang, Xianhao Xiong, Danyu Tang, Xinshuo Hu, Guohong Fu, Meishan Zhang, Baotian Hu
arXiv Computer Vision
Oct 12

Robust Vision-Language Navigation with Awareness of Instruction-Execution Progress

The paper introduces VLN-AIEP, a Vision‑Language Navigation framework that integrates instruction‑execution progress awareness into navigation decisions. By linking the visited trajectory to the instruction, it suppresses embeddings of already executed content and matches candidate locations with pending instruction segments. The method employs a landmark‑anchored trajectory representation to encode semantic labels and geometry, enabling progress estimation without explicit supervision and achieving strong results on R2R, RxR, CVDN, and R2R‑CE benchmarks.

By Jiagan Huang, Zikun Zhou, Hongpeng Wang, Guangming Lu, Jun Yu, Wenjie Pei