LMEnt is a toolkit designed to study how language models acquire and represent world knowledge. It includes a richly annotated pretraining corpus, an improved entity-based retrieval method, and twelve pretrained models with numerous checkpoints. The suite enables controlled experiments linking entity mentions in training data to downstream knowledge performance, revealing how co-occurrence patterns influence learning and editing difficulty.
By Daniela Gottesman, Alon Gilae-Dotan, Ido Cohen, Yoav Gur-Arieh, Marius Mosbach, Ori Yoran, Mor Geva
The paper introduces ElephantBench, a closed‑book knowledge probe with 1,094 multi‑account factual questions generated via an auditable graph‑based pipeline that pulls documents from a low‑exposure web corpus and identifies naturally occurring disagreements. Across 32 large language models, even the best model only recovers both divergent accounts on 52.4% of questions, and most models recall one account while omitting the other, indicating persistent epistemic myopia. The study shows that scaling model size and inference‑time reasoning improves recall but does not eliminate incompleteness, and that exposure imbalance in the corpus biases models toward the dominant account.
By Zhuoshi Pan, Junru Lu, Yan Qian, H. Vicky Zhao, Di Yin, Xing Sun
The paper introduces GONE, a benchmark for evaluating knowledge unlearning in large language models using structured knowledge graphs, and presents Neighborhood-Expanded Distribution Shaping (NEDS), a framework that leverages graph connectivity to separate forgotten facts from their semantic neighborhood. GONE disentangles direct fact removal, reasoning-based leakage, and catastrophic forgetting, while NEDS achieves high unlearning efficacy and locality on LLaMA-3-8B and Mistral-7B. The dataset is publicly available on Hugging Face.
By Chahana Dahal, Ashutosh Balasubramaniam, Zuobin Xiong
arXiv:2608. 20106v1 Announce Type: new Abstract: We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers.
By Nikita Khudov
arXiv:2606. 19625v2 Announce Type: replace-cross Abstract: We use training-data attribution as an interpretable tool for capability discovery, mapping which regions of the pretraining corpus support social-reasoning versus STEM-reasoning in OLMo3-7B.
By Glenn Matlin, Chandreyi Chakraborty, Saehee Eom, Mika Okamoto, Rayan Castilla, Louis Jaburi, Alvin Deng, Taywon Min, Lucia Quirke, Stella Biderman, Mark Riedl
The paper introduces a method for identifying the dominant knowledge source behind large language model (LLM) outputs, distinguishing between faithfulness violations (misuse of provided context) and factuality violations (errors in internal knowledge). A simple linear probe trained on hidden representations can reliably classify this source, and the authors present AttriWiki, a self‑supervised pipeline that generates labeled training data by prompting models to recall withheld entities or read them from context. Probes trained on AttriWiki achieve high Macro‑F1 scores across several models and datasets, generalize zero‑shot to a benchmark, and show that attribution mismatches can increase error rates by up to 70%.
"whyItMatters":"The study demonstrates that knowing the source of an LLM’s answer is crucial for effective mitigation of hallucinations, as attribution mismatches significantly raise error rates."
By Ivo Brink, Alexander Boer, Dennis Ulmer