arXiv Computation and Language

GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

GeopoliticalQA is a new benchmark for retrieval‑augmented question answering over 415 open‑source geopolitical and strategic intelligence reports, containing 5,525 document‑grounded question‑answer pairs that cover factual, inferential, comparative, and risk‑oriented queries. The authors propose a document‑aware generation pipeline that profiles reports before generating questions, and they validate dataset quality through expert annotation and an LLM‑as‑judge protocol. Experiments on the benchmark show that while retrieval performance is similar across models at the document level, chunk‑level retrieval is not driven by embedding‑model scale, and answer generation remains difficult even with gold evidence, especially for inferential and risk‑oriented questions.

arXiv Machine Learning

Oct 3

Retrieval from Within: An Intrinsic Capability of Attention-Based Models

The paper introduces INTRA, an attention-based encoder-decoder framework that retrieves directly from its own internal representations instead of using an external retriever. By having decoder attention query pre-encoded evidence chunks, INTRA unifies retrieval and generation, eliminating the typical mismatch seen in retrieval-augmented generation pipelines. Experiments on question-answering benchmarks show that INTRA outperforms strong engineered retrieval pipelines in both evidence recall and overall answer quality.

By Elad Hoffer, Yochai Blau, Edan Kinderman, Ron Banner, Daniel Soudry, Boris Ginsburg

rag benchmarks

arXiv AI

Jul 9

Healthier LLMs: Retrieval-Augmented Generation for Public Health Question Answering

arXiv:2607. 06641v1 Announce Type: cross Abstract: Large language models (LLMs) achieve promising results on medical question answering benchmarks, yet their use in public health is constrained by hallucinations and the rapid evolution of official guidance.

By Felix Feldman, Joshua Harris, Timothy Laurence, Leo Loman, Ollie Higgins, Fan Grayson, Poonam Soma, Bethany Pace-Bonello, Michael Borowitz, Toby Nonnenmacher

llms rag nlp benchmarks safety

arXiv AI

Aug 5

SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG

arXiv:2608. 03860v1 Announce Type: cross Abstract: We introduce SciRet, a compute-aware empirical study of retrieval-augmented generation for scientific question answering over CORD-19.

By Kaysarul Anas Apurba, Md. Hasibul Hasan, Rofiqul Alam Shehab, Asab Azad

rag nlp benchmarks

arXiv AI

Jun 12

Uncertainty-Aware Hybrid Retrieval for Long-Document RAG

arXiv:2606. 13550v1 Announce Type: new Abstract: Retrieval augmented generation (RAG) depends critically on the quality and granularity of retrieved evidence.

By Hoin Jung, Xiaoqian Wang

rag nlp benchmarks

arXiv AI

Jun 10

LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.

By Haonan Wang, Jiaxiang Liu, Yurong Liu, Austin Senna Wijaya, Tianle Zhou, Eden Wu, Yijia Chen, Wanting You, Reya Vir, Daniela Pinto, Grace Fan, Yusen Zhang, Juliana Freire, Eugene Wu

llms agents nlp benchmarks

arXiv AI

Jul 1

SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA

arXiv:2504. 07385v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness.

By Sher Badshah, Ali Emami, Hassan Sajjad

llms agents benchmarks

arXiv AI

Jul 8

Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

arXiv:2607. 06482v1 Announce Type: cross Abstract: Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings.

By So Hasegawa, Shailaja Keyur Sampat, Lei Liu, Wei-Peng Chen

llms agents benchmarks

arXiv Computation and Language

5d ago

The Bottleneck Is Retrieval, Not Reading: Diagnosing Evidence-Based Generation with RAG for Long Context Question Answering

arXiv:2610.26951v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) is the dominant paradigm for grounding large language models in external corpora, but its similarity-driven, chunk...

By Debopriyo Banerjee, Owais Aijaz, Ahmer Tabassum, Zhuohan Xie, Preslav Nakov, Veselin Stoyanov

llms rag nlp benchmarks

arXiv AI

Oct 1

Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering

The paper presents a controlled comparison of six retrieval-augmented generation (RAG) strategies for scientific question answering on a large arXiv corpus. All pipelines use the same LLM generator and evaluation protocol, differing only in retrieval design—ranging from classic dense retrieval to late‑interaction methods like ColBERTv2. The authors also release a synthetic question dataset and code to enable reproducible, large‑scale evaluation of RAG trade‑offs.

By Bhagyesh Rathi, Eshan Chawla, William B. Andreopoulos

llms rag agents nlp

arXiv AI

Jun 6

Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)

arXiv:2606. 05901v1 Announce Type: cross Abstract: Large language models (LLMs) have fundamentally transformed the landscape of Natural Language Processing.

By Christopher J. Wedge, Joshua Stutter, Danny Dixon, Jacek Ca{\l}a

llms rag agents nlp fine-tuning benchmarks safety

arXiv AI

Oct 16

CARA: Causally Guided Retrieval Agent for Question Answering

CARA is a training‑free, causally guided retrieval agent designed for question answering over text‑rich knowledge graphs. It evaluates retrieved evidence using two causal criteria—consequentiality and essentiality—alongside path consistency, and aggregates results from multiple agents via reciprocal rank fusion. Experiments on three STaRK datasets show that CARA outperforms existing retrievers, improving answer quality and faithfulness while examining fewer graph nodes and requiring fewer retrieval operations.

By Weicheng Liu, Xiangcheng Wu, Depeng Xu, Haopeng Zhang, Xi Niu

rag agents nlp

arXiv AI

Jun 4

Self-Evolving Deep Research via Joint Generation and Evaluation

arXiv:2606. 04507v1 Announce Type: cross Abstract: Large Language Models (LLMs) have become increasingly adopted in daily applications, with deep research standing out as a particularly important capability.

By Han Zhu, Chengkun Cai, Yuanfeng Song, Xing Chen, Sirui Han, Yike Guo

llms agents reinforcement-learning benchmarks

Related stories