The study examines whether large language models (LLMs) show syntactic convergence with human users by analyzing turn‑adjacent reuse of context‑free grammar (CFG) rules in 1,901 matched positions across sixteen open‑weight Llama and Gemma models. All models exhibited greater CFG‑rule overlap with the preceding human turn than with unrelated primes, especially for lower‑frequency rules, and instruction‑tuned models showed even higher overlap with the actual prime than the human responses they replaced. However, instruction tuning also increased overlap with unrelated primes, reduced the actual‑versus‑random difference, and lowered conditional rule‑reuse odds when controlling for rule‑set size, while consistently producing responses with higher lexical and semantic similarity to the preceding turn.
By Zandi Eberstadt
Instruction tuning is often thought to give language models a universal ability to follow instructions, but this study shows otherwise. By probing nine tasks across three models, the authors find that general probes reveal selective, not uniform, deficits, cross‑task transfer is weak and skill‑similar, and causal ablation uncovers sparse, asymmetric dependencies. The results suggest instruction following is a coordinated use of diverse linguistic skills rather than a single shared mechanism.
By Elisabetta Rocchetti, Alfio Ferrara
The paper investigates how large language models balance instruction-following with pattern completion when the two objectives conflict. By creating dialogues where a user instruction to act in a target way T is opposed by assistant turns that demonstrate a competing pattern P, the authors measure instruction-following rates across 13 models and 16 instructions over up to 50 turns. Results show wide variability (1%–99%) in instruction adherence, with robustness influenced by instruction content, output format, and chain-of-thought reasoning, but overall instruction-following remains brittle under induction pressure.
By Carolina Camassa, Derek Shiller
The paper introduces XIH-Bench, a multilingual benchmark for evaluating instruction hierarchy (IH) compliance in large language models across six languages, four domains, and three IH settings. It finds that IH compliance varies with language, showing an asymmetry where a language can strengthen compliance in a higher-priority role but disrupt it in a lower-priority role, and that cross-language conflicts yield higher compliance than same-language conflicts—a phenomenon called the Language Boundary Effect. The study also highlights that language specialization can make lower-priority instructions in favored languages harder to override, raising potential reliability and security concerns.
By Jiwon Moon, Yerin Hwang, Kyomin Jung
arXiv:2608. 12334v1 Announce Type: cross Abstract: Despite the impressive multilingual capabilities of Large Language Models, the latent dynamics dictating language selection remain poorly understood.
By Arnav Srivastav
The paper introduces Language Identity Head Ablation (LIHA), a causal method that zeroes individual attention heads in transformer models to measure language switch rates across multilingual prompts. Applying LIHA to GPT‑2 reveals a small set of first‑token broadcaster heads—most notably L6H1—that persistently attend to the initial prompt token and propagate language signals throughout generation, with compensatory head recruitment occurring hierarchically in higher layers. A controlled comparison between Qwen2.5‑1.5B‑Base and Qwen2.5‑1.5B‑Instruct shows that instruction tuning concentrates language‑identity influence in early layers, while experiments with Chinese and Russian confirm script‑specific first‑token broadcasting at layer 0.
By Arjun Pillai, Christian Hoang, Anjelo Jann Laroza