arXiv:2606. 07808v1 Announce Type: new Abstract: Reasoning language models deployed in agentic workflows must follow an instruction hierarchy: when instructions from different sources conflict, the model should obey the highest-privilege applicable instruction.
By Sanjay Kariyappa, G. Edward Suh
The paper introduces Many-Tier Instruction Hierarchy (ManyIH), a new framework for resolving conflicts among instructions with arbitrarily many privilege levels in large language model agents. It presents ManyIH-Bench, a benchmark featuring 853 agentic tasks that require navigating up to 12 levels of conflicting instructions across 46 real-world agents. Experiments show current models achieve only about 40% accuracy when instruction conflict scales, highlighting a gap in fine-grained, scalable conflict resolution.
By Jingyu Zhang, Tianjian Li, William Jurayj, Hongyuan Zhan, Benjamin Van Durme, Daniel Khashabi
arXiv:2608. 02639v1 Announce Type: cross Abstract: Production prompts rarely carry a single instruction.
By Atul Anand, Sourav Chattaraj
Large Language Models often fail to respect instruction hierarchies in multi-turn settings, sometimes following lower-priority directives over higher ones. The authors formalize this failure using a Jensen‑Shannon Divergence framework and introduce IHDec, a contrastive decoding method that detects hierarchy violations at the token level and suppresses subordinate role influence without any fine‑tuning. Experiments show IHDec outperforms training‑based baselines, maintains overall response quality, improves safety against adversarial prompts, and scales well with larger models.
By Nicole Geumheon Liu, Haeun Jang, Yonghyun Jun, Hwanhee Lee
arXiv:2510. 07315v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural language interactions until it passes their vibe check.
By Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Liu, Jiawei Han, Benoit Schillings, Jiao Sun
The paper introduces a fine-grained method called interactions to analyze prompt sensitivity in large language models (LLMs). By decomposing output scores into nonlinear interactions, the authors show that subtle prompt changes can destabilize these interactions even when overall outputs stay unchanged. They propose an Interaction-based Prompt Sensitivity (IPS) metric and use it to evaluate 50 open-source LLMs, finding that supervised fine‑tuning, larger model scales, dense architectures, and few‑shot learning all reduce prompt sensitivity, primarily by stabilizing low‑order interactions.
By Ruiyang Qin, Qingzhuo Wang, Tian Wang, Zhihua Wei, Wen Shen