arXiv AI By Md. Rejaul Korim Sadi, Toufiqur Rahman Tasin, Golam Mostofa Naeem

From Architecture to Output: Testing the Structural Origins of Hallucination and the Amplifying Role of Data

Read the original on arXiv AI →

The paper investigates whether hallucinations in language models stem from their architecture or from the data they are trained on. By swapping attention mechanisms for state‑space mixers in Pythia/Mamba models, the authors show that attention models are more prone to repeating false statements, while state‑space models better handle planted facts. A second study demonstrates that correcting a single erroneous fact in instruction‑tuned models significantly reduces downstream errors, indicating that the structure of the model influences how errors propagate.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv AI
Oct 16

Language Models Share a Readout Bottleneck in Working Memory

Large language models (LLMs) exhibit a human‑like working‑memory pattern on the N‑back task, with performance dropping as memory load increases and errors mainly on lure items. Two hypotheses—precision limits of softmax attention and interference from competing items—were tested across 15 pretrained LLMs and minimal transformers. Results show that interference, not precision, constrains LLMs’ working‑memory performance, as suppressing non‑target item identities improves sensitivity and larger models benefit more from reducing interference.

By Dongyu Gong