arXiv Computation and Language By Shaswat Patel, Vishvesh Trivedi, Yue Han, Yihuai Hong, Eunsol Choi

Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads

Read the original on arXiv Computation and Language →

The paper investigates attention heads in multilingual Transformer models, distinguishing between retrieval heads that pull information from context and a newly identified class called Retrieval‑Transition heads (RTH) that direct the model toward a specific target language. Experiments across four multilingual benchmarks and two model families show that masking RTHs causes a larger performance drop than masking retrieval heads, indicating RTHs are crucial for chain‑of‑thought reasoning in multilingual LLMs. The study thus clarifies which attention heads are responsible for mapping to target languages, advancing our understanding of multilingual language models.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Computation and Language.

arXiv Computation and Language
Oct 5

Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation

The study explores how the language used for reasoning affects retrieval‑augmented generation (RAG) in a monolingual German setting. Using a German RAG question‑answering testbed based on the tabletop game The Dark Eye, the authors show that aligning the reasoning language with the query and retrieved documents improves performance, with German reasoning outperforming French reasoning. However, German reasoning still does not surpass the model’s native English reasoning, indicating that native multilingual reasoning is necessary for optimal results.

By Oliver Hauck, Mario Sanz-Guerrero, Katharina von der Wense
arXiv AI
Oct 26

Head Dimension Matters in Language Models for Long-Context Retrieval

The paper investigates how the dimension of attention heads influences long‑context retrieval in language models. Experiments show that while pretraining loss and short‑context tasks are largely unaffected, wider heads significantly improve retrieval performance at fixed model sizes. Ablation studies indicate that this benefit persists even when attention weights are frozen, suggesting that wider heads enhance adaptation of other model components such as embeddings and MLPs.

By Seiya Tokui, Shoichiro Yamaguchi, Sosuke Kobayashi