← Back to all news
arXiv Machine Learning By Aman Behera, Namit Solanki, Mehul Anand

Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

Read the original on arXiv Machine Learning →

The study investigates how the position of evidence documents relative to a prompt affects long‑context question answering across different sequence mixers—Transformer, state‑space, and hybrid Mamba models. Results show that Transformers exhibit a beginning‑position advantage that state‑space models lack, while both families tend to favor evidence near the end. Additional experiments reveal that prompt design, corpus choice, and model capability influence performance, and that position bias remains decodable across architectures even when response curves differ.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Sep 3

Language Models Can Control Their Own Attention

The paper introduces Declarative Attention (DA), a protocol that lets language models explicitly declare which parts of their context to focus on during generation. By partitioning decoding into full-context, region-specific, and recent-output-only modes, the inference engine can skip large portions of the KV cache, dramatically reducing attended tokens. Experiments on 15 long-context tasks with off-the-shelf models show significant savings (52.0% and 31.1% reductions) with only modest accuracy drops that diminish as model size increases.

By Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, Cicero Nogueira dos Santos
llmsefficiency
More like this →
arXiv AI
Sep 30

Locating Answer-Correctness Signals in Frozen Large Language Models

The paper investigates where and how large language models encode signals that indicate answer correctness. By examining hidden states, token probabilities, residual-stream features, attention, and their combinations, the authors find that correctness signals are concentrated in the answer span and that different signal families complement each other. Fusing these signals improves robustness, especially under distribution shifts, and can be used to control retrieval in downstream tasks.

By Yuansen Liu, Yixuan Tang, Anthony Kum Hoe Tung
llmsrag
More like this →