arXiv Computation and Language

Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge

The paper introduces ElephantBench, a closed‑book knowledge probe with 1,094 multi‑account factual questions generated via an auditable graph‑based pipeline that pulls documents from a low‑exposure web corpus and identifies naturally occurring disagreements. Across 32 large language models, even the best model only recovers both divergent accounts on 52.4% of questions, and most models recall one account while omitting the other, indicating persistent epistemic myopia. The study shows that scaling model size and inference‑time reasoning improves recall but does not eliminate incompleteness, and that exposure imbalance in the corpus biases models toward the dominant account.

arXiv Computation and Language

3d ago

Agent Zero Memory: Provenance-Aware Long-Term Memory for LLM Agents

Agent Zero Memory is a provenance‑aware long‑term memory system for large language model agents that distills user interactions into three parallel memory structures: an episodic timeline, an associative entity‑event knowledge graph, and a semantic, citation‑locked hierarchical documentary memory. Retrieval is performed via an intent gate, source router, and concurrent searches across the three systems, producing integrated, cited answers that exclude fabrication and require evidence the reader has opened. The system achieves state‑of‑the‑art performance on LongMemEval (95.60%) and LoCoMo (93.60%) while offering a favorable accuracy‑cost‑latency trade‑off across multiple backbone LLMs.

By Ming Wu, Pengyuan Zhu

llms rag agents benchmarks

arXiv AI

Jul 8

Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

arXiv:2607. 06482v1 Announce Type: cross Abstract: Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings.

By So Hasegawa, Shailaja Keyur Sampat, Lei Liu, Wei-Peng Chen

llms agents benchmarks

arXiv AI

Jun 10

LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.

By Haonan Wang, Jiaxiang Liu, Yurong Liu, Austin Senna Wijaya, Tianle Zhou, Eden Wu, Yijia Chen, Wanting You, Reya Vir, Daniela Pinto, Grace Fan, Yusen Zhang, Juliana Freire, Eugene Wu

llms agents nlp benchmarks

arXiv AI

Jul 14

CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps

arXiv:2607. 10562v1 Announce Type: new Abstract: Evaluating the multi-hop reasoning capabilities of large language models remains a significant challenge.

By JungMin Yun, JuneHyoung Kwon, YoungBin Kim

llms nlp

arXiv Computation and Language

Aug 21

OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models

arXiv:2608. 20106v1 Announce Type: new Abstract: We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers.

By Nikita Khudov

llms agents benchmarks

arXiv Computation and Language

2d ago

LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs

arXiv:2609.01182v1 Announce Type: new Abstract: Flagship language models appear saturated on benchmarks like MMLU (Hendrycks et al., 2021), scoring above 90% - yet benchmarks test only what the exper...

By Muhammed Saeed, Simon Razniewski

llms benchmarks safety

arXiv Computation and Language

Aug 25

Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

arXiv:2608.22856v1 Announce Type: cross Abstract: A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval pol...

By Jingjie Ning, Xueqi Li

rag agents

arXiv AI

Jun 18

RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositories

arXiv:2512. 04144v2 Announce Type: replace Abstract: Targeted interventions on language models, such as unlearning or model editing, aim to modify specific information, but their effects often propagate to related, unintended areas (e.

By Roy Rinberg, Usha Bhalla, Igor Shilov, Flavio P. Calmon, Rohit Gandikota

llms rag

arXiv AI

Jul 23

Information Discernment in Large Language Models

arXiv:2607. 19355v1 Announce Type: new Abstract: LLMs are increasingly used with external knowledge sources like the internet.

By Joshua Ashkinaze, Laura Kurek, Alina Faisal, Tongyuan Miao, Mariam Joseph, Ceren Budak, Eric Gilbert

llms benchmarks safety

arXiv AI

Aug 11

SodaMem: Evidence-Grounded Temporal Graph Memory for LLM Agents

arXiv:2608. 08055v1 Announce Type: new Abstract: Large language model (LLM) agents that assist users over weeks of conversation must remember what is currently true, not merely what was once said.

By Fengrong Wan, Chengcan Wu, Ningtao Lyu

llms rag agents

Hugging Face Trending Papers

Aug 11

ENTLORE: A Graph-Grounded Benchmark for Latent Organizational Reasoning in Enterprise Question Answering

Enterprise question answering is framed as retrieving internal documents and generating grounded answers. Routine enterprise records, however, are work by-products in which required organizational relations remain implicit across heterogeneous sources.

rag nlp benchmarks

arXiv Computation and Language

4d ago

Entity-Memory Graph Retrieval Improves Evidence Coverage in Long-Conversation Question Answering

Entity-Memory graph retrieval preserves dialogue turns as verbatim memory nodes, links repeated mentions via shared entities, and connects adjacent memories with chronological edges. During retrieval, the system gates through entities, fuses semantics, and performs one‑hop chronological recovery before dense backfill, allowing it to keep neighboring memories that dense cosine ranking might miss. On 1,986 questions from ten LoCoMo conversations, this graph retrieval method increases official evidence recall at top‑k 25 from 79.7468 % to 84.4842 %, with the advantage extending from top‑k 5 to 50, though it does not improve overall final‑answer F1.

By Shumao Sun

llms rag nlp

Related stories