← Back to all news
arXiv Computation and Language By Abhinav Havaldar, Enrico Santus

When RAG Fails to Equalize: Geo-bias in Factual Question Answering over Public Companies

Read the original on arXiv Computation and Language →

The paper investigates whether retrieval‑augmented generation (RAG) can uniformly correct factual errors in large language models (LLMs) by testing six LLMs on a benchmark of about 2,000 public companies. In a controlled factual QA setting, the authors evaluate the models under four conditions—no‑context, perfect context, misleading context, and distraction context—across four atomic attributes. Results show significant geographic disparities in baseline accuracy, and while perfect context improves performance, it does not eliminate these gaps; misleading context often leads models to copy incorrect information, and larger models only marginally reduce structural biases.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

Related stories

arXiv Computation and Language
Sep 10

Do LLMs Make More Mistakes If They Do Not Believe the Input Data?

The study investigates whether large language models (LLMs) are more prone to errors when they doubt the plausibility of input data, a phenomenon termed context‑memory conflict. Using non‑English and low‑resource language datasets, the authors generate text from factual, counterfactual, and fictional RDF triples in English, Czech, Slovak, and Upper Sorbian, and evaluate faithfulness with both human annotations and an LLM judge (Kimi K3). Contrary to expectations, the results show only a weak context‑memory conflict: counterfactual inputs receive slightly lower faithfulness scores than factual ones, and the choice of LLM judge can significantly affect perceived conflict strength.

By Peter Kochelka, Ale\v{s} Manuel Pap\'a\v{c}ek, Vojt\v{e}ch Dvo\v{r}\'ak, Ond\v{r}ej Du\v{s}ek
llmsrag
More like this →
arXiv AI
Jun 24

Quantifying Prior Dominance in RAG Systems

arXiv:2606. 23695v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) grounds Large Language Models in external knowledge, yet current evaluations rely on discrete heuristics that suffer from ''epistemic blindness'' - failing to distinguish genuine contextual information extraction from parametric memory recall.

By Barak Or
llmsragsafety
More like this →
arXiv AI
Sep 15

Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain

The paper evaluates seven open‑source large language models for retrieval‑augmented generation in the ESG reporting domain, using 498 real‑world ESG reports from EU‑listed companies and 100 synthetic QA pairs. Performance is measured with RAGAS metrics, showing strong retrieval scores but variable generation quality, especially in faithfulness and factual correctness. The results highlight significant differences across model architectures and underscore the need for domain‑specific fine‑tuning to improve factual accuracy.

By Motaz Saad, Anna Borrelli, Ivan Gentile, Kianna Kazemi, Francesco Piccialli, Antonella Longo
llmsragfine-tuning
More like this →
arXiv AI
Sep 10

No\=esis: Deterministic-First Retrieval with Two-Tier Context Hydration for Factuality-Critical Queries on Small Local Models

Noesis is a retrieval-augmented architecture designed for small local language models that prioritizes deterministic judgments before generation. It employs a fact layer, positional addressing, provenance scoping, and two-tier context hydration to ensure factual integrity, achieving parity with larger models on exact value accuracy and eliminating confabulated numbers. The system delivers single-generation queries with traceable, source‑linked outputs, improving performance in regulated domains.

By Nicola Cogotti
llmsragsafety
More like this →