arXiv AI

Words Speak Louder Than Order: A Behavioral Evaluation of Gemma 4

The study investigates how Google’s Gemma 4‑e4b language model resolves conflicts between two documents. Using a counterbalanced design, researchers found that the semantic framing of a source (e.g., labeling it as an official guideline) dominates over the order in which documents appear. While the model shows a primacy bias toward the first document, this bias varies widely with wording and is amplified only when the documents are structurally identical.

arXiv Computation and Language
Sep 18

Summarization Bias: The Directional Collapse of Objective Projection into Told-Mode Labels in Large Language Models --- A Conceptual Framework and Registered Test Protocol

The paper introduces the concept of summarization bias in large language models (LLMs), describing a systematic tendency for LLMs to represent narrative meaning as an abstract summary label rather than the reconstructable inferential structure that produces it. It frames this bias within the Bulut Doctrine’s told‑shown axis, arguing that LLMs fail in a specific direction: they default to told‑mode explicitness in generative tasks and reward told‑mode explicitness while under‑detecting shown‑mode suppression in evaluative tasks. The authors outline two regimes of bias, present preliminary evidence, and pre‑register a test protocol to validate or abandon the construct.

By Levent Bulut
arXiv Machine Learning
Oct 23

Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

The study investigates how the position of evidence documents relative to a prompt affects long‑context question answering across different sequence mixers—Transformer, state‑space, and hybrid Mamba models. Results show that Transformers exhibit a beginning‑position advantage that state‑space models lack, while both families tend to favor evidence near the end. Additional experiments reveal that prompt design, corpus choice, and model capability influence performance, and that position bias remains decodable across architectures even when response curves differ.

By Aman Behera, Namit Solanki, Mehul Anand
arXiv Computation and Language
Nov 4

Instruction-Tuned Models Locally Produce More Syntactic Overlap -- But Not More Conditional Reuse -- Than Humans Do

The study examines whether large language models (LLMs) show syntactic convergence with human users by analyzing turn‑adjacent reuse of context‑free grammar (CFG) rules in 1,901 matched positions across sixteen open‑weight Llama and Gemma models. All models exhibited greater CFG‑rule overlap with the preceding human turn than with unrelated primes, especially for lower‑frequency rules, and instruction‑tuned models showed even higher overlap with the actual prime than the human responses they replaced. However, instruction tuning also increased overlap with unrelated primes, reduced the actual‑versus‑random difference, and lowered conditional rule‑reuse odds when controlling for rule‑set size, while consistently producing responses with higher lexical and semantic similarity to the preceding turn.

By Zandi Eberstadt
arXiv Computation and Language
Oct 22

Linguistic Productivity in Large Language Models: Models Coerce, but do not Preempt

The paper investigates how large language models (LLMs) handle two usage‑based grammatical strategies: coercion, where a mismatched word is reinterpreted to fit its context, and preemption, where a plausible structure is avoided because it never appears in expected contexts. Experiments across models of varying sizes show that larger LLMs successfully recognize and produce constructional productivity in coercive situations, even with novel words, but they fail to exhibit preemptive behavior, lacking the negative evidence that would guide avoidance of unconventional but semantically viable constructions. The authors argue that coercion is detectable due to its surprising, semantically motivated nature, whereas preemption lacks such a signature, leading to an asymmetry where models coerce but do not preempt.

By Claire Bonial, Claire Benet Post, Laura Michaelis, Harish Tayyar Madabushi
arXiv AI
Sep 24

Reporting Under Pressure: Separating Factual and Tonal Sycophancy in LLM Statistical Analysis

The study examines how different editorial framings in prompts influence large language models’ statistical analysis reports. Using a 4×4 factorial design, researchers found that certain framings—particularly brutally critical prompts on genuine effects and significance-seeking prompts on underpowered nulls—led to factual misrepresentations. Tone shifts were more widespread, with critical framing inducing defensive language across all data patterns, while a confound in the data largely prevented both factual and tonal distortions.

By Paras Balani, Subhrakanta Panda