Retrieval-augmented generation

Retrieval pipelines, vector search, chunking and reranking: how models are grounded in a corpus instead of their weights.

6,383 stories · RSS feed

arXiv AI
Aug 24

PrimeAgentOrchestrator: Memory-Primed Agent Spawning for Personal AI Infrastructure

PrimeAgentOrchestrator (PAO) is a system that spawns new instances of Claude Code, Anthropic’s terminal-based coding agent, pre‑loaded with relevant memories from a user’s personal databases. PAO queries a PostgreSQL entity‑observation database and a Cloudflare Worker semantic search index in parallel, fuses the results, and injects the compiled briefing into the agent’s filesystem to exploit its auto‑read configuration. The paper reports four months of deployment, detailing three generations of context delivery mechanisms, the failure modes that prompted each redesign, and the engineering tradeoffs involved in integrating heterogeneous memory systems instead of building a unified one.

By Myron Koch (Peak Summit Labs)
arXiv Computer Vision
Aug 24

Reason-Then-Retrieve for CoVR-R with Structured Edit Prompts and Dense-Sparse Fusion

The paper introduces CoVR‑R, a reason‑aware composed video retrieval system that, given a reference video and an edit instruction, retrieves a target video that satisfies the edit. It employs a zero‑shot reason‑then‑retrieve pipeline using Qwen3.5‑27B to generate structured descriptions and dense embeddings for gallery videos, and performs edit reasoning on the query to produce a target‑video description used as the query embedding. The method combines dense retrieval with a TF‑IDF branch over generated texts, fusing the rankings with split‑specific weights, achieving state‑of‑the‑art retrieval metrics on both validation and blind test splits.

By Dongqing Liu, Mengshi Qi, Hongwei Ji
arXiv AI
Aug 24

Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems

The paper introduces Trustworthy RAG, an evaluation agent designed to detect misinformation and knowledge poisoning in Retrieval-Augmented Generation systems. It combines natural language inference verification, a five-signal poison detector, and a weighted Trust Index to assess the reliability of retrieved content. Experiments on multiple LLMs show high accuracy and precision, with the agent effectively blocking unsafe advice in a secure-coding assistant scenario.

By Balkrishna Giri, Md Toufique Hasan, Jussi Rasku, Muhammad Waseem, Pekka Abrahamsson
arXiv Computation and Language
Aug 24

Granuscore: A Reference-Free Measure of Granularity for Text Analysis and Question Answering

Granuscore is a reference‑free metric that measures the granularity of text by exploiting the structure of a hierarchical embedding space. It successfully reproduces known hierarchical orderings on the Granola‑EQ dataset, distinguishes granularity across different discourse contexts, and explains sentence‑specificity variations beyond sentence length. The authors also apply Granuscore to four question‑answering benchmarks, revealing systematic differences in granularity among questions, gold answers, and model outputs, thereby offering a new lens for assessing QA dataset difficulty.

By Lukas Ellinger, Alexander Fichtl, Miriam Ansch\"utz, Georg Groh
arXiv Computer Vision
Aug 24

ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval

arXiv:2608.15698v2 Announce Type: replace Abstract: Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from docume...

By Chunyi Peng, Zhipeng Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Sen Mei, Yubo Sun, Yongheng Zhang, Jie Zhou, Yu Gu, Ge Yu, Maosong Sun
Towards Data Science
Aug 23

Parse the Folder, Not Just the PDFs: The Relational Tables RAG Needs on a Case File

The article discusses how Enterprise Document Intelligence should begin by parsing the folder structure rather than just PDFs, emphasizing that the index must reflect the case type’s requirements before any folder is accessed. It highlights that the two key questions to develop are not retrieval questions but rather focus on the relational tables needed for Retrieval-Augmented Generation (RAG) in a case file context.

By angela shi
Simon Willison
Aug 22

llm 0.33

The release of llm 0.33 introduces several key updates: it upgrades to the OpenAI Python library 3.x and switches the HTTP client from httpx to httpx2, adds comprehensive key handling for embedding functions, allows repeated template usage to combine configurations, and adds a reasoning_summary option for reasoning-capable responses. These changes improve compatibility, flexibility, and usability of the llm tool.

Towards Data Science
Aug 22

Multi-Document RAG: A Folder of Unrelated PDFs Is One Long Document with a Nested Outline

The article discusses a method for handling a folder of unrelated PDFs as a single long document with a nested outline. It highlights that without shared fields, an index cannot be built, so the approach uses one summary line per file and each file’s own table of contents, with retrieval routes extending down two levels. This structure enables retrieval-augmented generation (RAG) across multiple documents.

By angela shi
arXiv Machine Learning
Aug 21

Decoding silent reading from non-invasive EEG

arXiv:2608. 20186v1 Announce Type: new Abstract: Non-invasive decoding of inner speech faces a fundamental data problem: a corpus pairing brain activity with a person's spontaneous inner monologue cannot be collected, and the available proxy paradigms (cued repetitive and retrospectively reported generative inner speech) are slow to acquire, poorly time-locked, and subject compliance is unverifiable.

By Ingo Marquardt, Anthilia Alchanat, Priyanka Jain
arXiv Computation and Language
Aug 21

Disentangling Speaker Traits for Deepfake Source Verification via Chebyshev Polynomial and Riemannian Metric Learning

arXiv:2603. 21875v2 Announce Type: replace-cross Abstract: Speech deepfake source verification systems aims to determine whether two synthetic speech utterances originate from the same source generator, often assuming that the resulting source embeddings are independent of speaker traits.

By Xi Xuan, Wenxin Zhang, Zhiyu Li, Jennifer Williams, Ville Hautam\"aki, Tomi H. Kinnunen