GeopoliticalQA: Benchmarking Retrieval and Generation over Intelligence Reports

GeopoliticalQA is a new benchmark for retrieval‑augmented question answering over 415 open‑source geopolitical and strategic intelligence reports, containing 5,525 document‑grounded question‑answer pairs that cover factual, inferential, comparative, and risk‑oriented queries. The authors propose a document‑aware generation pipeline that profiles reports before generating questions, and they validate dataset quality through expert annotation and an LLM‑as‑judge protocol. Experiments on the benchmark show that while retrieval performance is similar across models at the document level, chunk‑level retrieval is not driven by embedding‑model scale, and answer generation remains difficult even with gold evidence, especially for inferential and risk‑oriented questions.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

• llms
• rag
• nlp
• benchmarks