Public institutions hold large volumes of sensitive documents and support tickets that cannot leave the premises, ruling out cloud-hosted language models entirely. We report on RAGAL, a retrieval-augmented assistant for the technical-support team of AFIR, the Romanian Agency for Financing Rural Investments, built and operated under three hard constraints: zero data egress (no external API calls, even for synthetic data), a read-only mandate (the assistant drafts, humans execute), and a single 8 GB consumer laptop as the only development and training machine.
Legal Research Bench (LRB) is a new benchmark comprising 413 open-ended U.S. legal research questions, each paired with a gold answer, supporting authorities, and a binary grading rubric. The study evaluates thirteen advanced language‑model agents using web search, case‑law search, page parsing, and retrieval tools, scoring responses only when all required criteria are met and cited authorities verify. Results show that even the best model, Claude Opus 4.8, achieves full correctness on only 42.9% of questions, with performance varying by legal area and task complexity, and no clear link between more tool calls or inference cost and higher accuracy.
By Katrina Drozdov, Oliver Chen, Langston Nashold, Rayan Krishnan
arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.
By Haonan Wang, Jiaxiang Liu, Yurong Liu, Austin Senna Wijaya, Tianle Zhou, Eden Wu, Yijia Chen, Wanting You, Reya Vir, Daniela Pinto, Grace Fan, Yusen Zhang, Juliana Freire, Eugene Wu
Q2D-Web is a new large‑scale benchmark for agentic Retrieval‑Augmented Generation (RAG) systems, featuring a 190 million‑document web corpus and 70 k machine‑reformulated search queries in ten languages. It supplies three sets of relevance judgments—agent citations, production rankings, and a combined set enriched with LLM‑based labels—to evaluate first‑stage retrievers. Experiments on 13 retrievers show consistent ranking across judgment sets but significant variation across domains, languages, and query types, and demonstrate that a carefully sampled sub‑corpus can approximate full‑corpus evaluation with minimal loss in Recall@1000.
The paper presents a smartphone‑compatible, retrieval‑augmented language model tailored to Bangladeshi statutory law. By distilling a 9‑billion‑parameter Gemma‑2 teacher into a 2‑billion‑parameter student using supervised fine‑tuning and QLoRA, the authors achieve significant gains in ROUGE‑L and BERTScore on an English benchmark while keeping the model lightweight (1.6 GB) and operable offline on a Pixel 6. The system retrieves from 36,029 statutory passages using a hybrid dense/BM25 approach, and cross‑lingual evaluation shows effective Bangla query handling against an English‑only corpus, with a practicing lawyer rating the responses highly in a pilot study.
By MD. Nafis Kamal, Mahadi Hasan Fahim, Talha Ridwan, Nadifa Zaman, Fariha Roushon Florin, Farig Yousuf Sadeque, Saadat Rafid Ahmed
Clarify-Then-Search is a benchmark that tests whether large language models can ask clarification questions to improve the usefulness of deep search results. It uses 518 real-world query pairs from Baidu, where each intent query is paired with an underspecified version. The evaluation involves a clarifier asking up to three questions, a user answerer providing only explicit information, and a rewriter generating a new query that is then searched; performance is measured by a weighted nugget-recall score.
By Deqiang Huang, Jingbo Zhou, Xinjiang Lu, Tong Xu, Hua Wu, Enhong Chen
arXiv:2608. 12282v1 Announce Type: new Abstract: Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation.
By Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder, Siyu Huo, Raavi Gupta, Abhinav Jain, Praveen Venkateswaran, Abdulhamid Adebayo, Danish Contractor
arXiv:2606.11350v2 Announce Type: replace
Abstract: Retrieval-augmented generation (RAG) degrades when applied to large, heterogeneous document collections. In RAG systems, dense similarity becomes l...
By Nabaraj Subedi, Ahmed Abdelaty, Shivanand Venkanna Sheshappanavar
arXiv:2610.14428v1 Announce Type: new
Abstract: Enterprise workflow automation typically requires retrieving a complete and accurate set of relevant information from large operational document corpor...
By Istok Bako, Bonan Min, Tiffany Shao, Claire Cheng, Yashar Mehdad
The paper presents a deployed system for answering questions over normative documents that is aware of document version and scope. It evaluates a hosted retrieval service against a governed system that applies explicit rules for version and scope resolution, finding the governed system achieves a higher score (97.7 vs 88.1). The study includes a public benchmark, evaluation scripts, and reports commercial deployment metrics, such as 1,126 users and 100,000 calls per day by April 2026.
By Liuyin Wang, Shuaipeng Jin, Jiwei Shi, Jensen Hsu
CacheGraft is a retrieval‑augmented generation system that enriches a small language model’s latent space by injecting fused key‑value cache representations of corpus passages, encoded offline by a larger model. The approach eliminates the need to run the larger model at inference time, and on five QA benchmarks it improves answer quality in 29 of 30 comparisons, with 17 statistically significant gains. These results demonstrate that latent‑space document enrichment can boost RAG performance, especially when only a small model can be deployed.
By Vihan Lakshman, Peter Baile Chen, Zhuohan Gu, Michael Cafarella
CanLegalRAGBench is a Canadian legal QA benchmark that uses realistic queries and expert‑annotated answers grounded in case law to evaluate retrieval‑augmented generation (RAG) systems. The study finds that retrieval performance varies with design choices and that open‑source embedding models can compete with closed‑source ones. It also highlights limitations of automatic evaluations, noting that RAG systems often produce hallucinations or overly detailed responses, with 8‑29% of claims unsupported by retrieved documents.
By Ethan Zhao, Maksym Taranukhin, Wei Cui, Moira Aikenhead, Vered Shwartz