arXiv Computation and Language

Blind Men and the Elephant: Probing the Epistemic Myopia of LLMs under Long-Tail Divergent Knowledge

The paper introduces ElephantBench, a closed‑book knowledge probe with 1,094 multi‑account factual questions generated via an auditable graph‑based pipeline that pulls documents from a low‑exposure web corpus and identifies naturally occurring disagreements. Across 32 large language models, even the best model only recovers both divergent accounts on 52.4% of questions, and most models recall one account while omitting the other, indicating persistent epistemic myopia. The study shows that scaling model size and inference‑time reasoning improves recall but does not eliminate incompleteness, and that exposure imbalance in the corpus biases models toward the dominant account.

arXiv Computation and Language

Sep 1

Agent Zero Memory: Provenance-Aware Long-Term Memory for LLM Agents

Agent Zero Memory is a provenance‑aware long‑term memory system for large language model agents that distills user interactions into three parallel memory structures: an episodic timeline, an associative entity‑event knowledge graph, and a semantic, citation‑locked hierarchical documentary memory. Retrieval is performed via an intent gate, source router, and concurrent searches across the three systems, producing integrated, cited answers that exclude fabrication and require evidence the reader has opened. The system achieves state‑of‑the‑art performance on LongMemEval (95.60%) and LoCoMo (93.60%) while offering a favorable accuracy‑cost‑latency trade‑off across multiple backbone LLMs.

By Ming Wu, Pengyuan Zhu

llms rag agents benchmarks

arXiv AI

Jul 8

Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

arXiv:2607. 06482v1 Announce Type: cross Abstract: Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings.

By So Hasegawa, Shailaja Keyur Sampat, Lei Liu, Wei-Peng Chen

llms agents benchmarks

arXiv AI

Jun 10

LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

arXiv:2606. 10460v1 Announce Type: cross Abstract: Recent large language models (LLMs) have shown rapid progress in reading-based question answering (QA), where evidence is explicitly provided or can be trivially retrieved.

By Haonan Wang, Jiaxiang Liu, Yurong Liu, Austin Senna Wijaya, Tianle Zhou, Eden Wu, Yijia Chen, Wanting You, Reya Vir, Daniela Pinto, Grace Fan, Yusen Zhang, Juliana Freire, Eugene Wu

llms agents nlp benchmarks

arXiv AI

Jul 14

CRiT-QA: Evaluating Multi-hop Reasoning with Counterfactual Chains and Distractor Traps

arXiv:2607. 10562v1 Announce Type: new Abstract: Evaluating the multi-hop reasoning capabilities of large language models remains a significant challenge.

By JungMin Yun, JuneHyoung Kwon, YoungBin Kim

llms nlp

arXiv Computation and Language

Aug 21

OenoBench: A Wine-Domain Benchmark for Knowledge-Grounded Evaluation of Large Language Models

arXiv:2608. 20106v1 Announce Type: new Abstract: We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers.

By Nikita Khudov

llms agents benchmarks

arXiv Computation and Language

Oct 7

Wikidata Search Traces: A Dataset for Training Knowledge Graph Search Agents

arXiv:2610.06650v2 Announce Type: replace Abstract: Wikidata is one of the largest open knowledge bases, yet answering a complex question over it still requires a SPARQL query that names the right en...

By Mohamed Chenene, Carlos Rosas-Hinostroza, Anastasia Stasenko, Shani Evenstein Sigalov, Pierre-Carl Langlais

llms agents

arXiv Computation and Language

Sep 2

LLMPEDIA: Browsing, Verifying, and Comparing the Parametric Encyclopedic Knowledge of LLMs

arXiv:2609.01182v1 Announce Type: new Abstract: Flagship language models appear saturated on benchmarks like MMLU (Hendrycks et al., 2021), scoring above 90% - yet benchmarks test only what the exper...

By Muhammed Saeed, Simon Razniewski

llms benchmarks safety

arXiv Computation and Language

Aug 25

Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

arXiv:2608.22856v1 Announce Type: cross Abstract: A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval pol...

By Jingjie Ning, Xueqi Li

rag agents

arXiv AI

Jun 18

RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositories

arXiv:2512. 04144v2 Announce Type: replace Abstract: Targeted interventions on language models, such as unlearning or model editing, aim to modify specific information, but their effects often propagate to related, unintended areas (e.

By Roy Rinberg, Usha Bhalla, Igor Shilov, Flavio P. Calmon, Rohit Gandikota

llms rag

arXiv AI

5d ago

Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models

arXiv:2506.17585v4 Announce Type: replace Abstract: Large language models can answer questions correctly and still hallucinate their citations. Most citation systems address this by retrieving docume...

By Yukun Huang, Sanxing Chen, Jian Pei, Manzil Zaheer, Bhuwan Dhingra

llms fine-tuning benchmarks

arXiv AI

Jul 23

Information Discernment in Large Language Models

arXiv:2607. 19355v1 Announce Type: new Abstract: LLMs are increasingly used with external knowledge sources like the internet.

By Joshua Ashkinaze, Laura Kurek, Alina Faisal, Tongyuan Miao, Mariam Joseph, Ceren Budak, Eric Gilbert

llms benchmarks safety

arXiv AI

Aug 11

SodaMem: Evidence-Grounded Temporal Graph Memory for LLM Agents

arXiv:2608. 08055v1 Announce Type: new Abstract: Large language model (LLM) agents that assist users over weeks of conversation must remember what is currently true, not merely what was once said.

By Fengrong Wan, Chengcan Wu, Ningtao Lyu

llms rag agents

Related stories