arXiv:2608.24621v2 Announce Type: replace
Abstract: Can language models be trusted in safety- critical operations? In such settings, strong per- formance on semantic metrics does not guaran- tee oper...
By Yujing Chang, Thinh Pham, Van-Phat Thai, Chunyao Ma, Yash Guleria, Pham Nhut Huy, Sameer Alam
arXiv:2607. 01829v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing assistants.
By Alex Brooker, Tim Hughes
DialFork is a new benchmark that evaluates procedural hallucination in conversational agents, a failure mode where agents produce fluent, factually plausible utterances but deviate from the intended dialogue process. The benchmark consists of 36,278 dialogues across 24 tasks in 12 domains, created by forking conformant dialogues and injecting one of four hallucination types, with validation by 30 human annotators. Three evaluation tasks are defined—monitoring overall deviation, procedural retrieval of a single dialogue, and real‑time detection of off‑track turns—using dialog‑flow‑aware metrics FlowKL and FlowScore, which outperform baseline semantic similarity metrics.
By Sergio Burdisso, Esa\'u Villatoro-Tello, Dairazalia S\'anchez-Cort\'es, Yanis Labrak, Jordi Luque, Milos Cernak, Petr Motlicek
arXiv:2606. 17904v1 Announce Type: new Abstract: Language models increasingly serve as advisory systems in maintenance operations.
By Guillermo Gil de Avalle, Laura Maruster, Shaina Raza, Christos Emmanouilidis
The study investigates how language models acquire in-context learning (ICL) by distinguishing between copying a nearby demonstration and integrating information across a prompt. Across 17 models, roughly one-third default to copied answers, a few are undecided, and the remainder favor rule-consistent answers that yield higher accuracy and are guided by the requested value rather than the nearest conflicting example. Analyses show that models prioritize task-relevant prompt tokens before accuracy surpasses chance, and that synthetic or fine-tuned training can shift models toward compositional responses or improve accuracy when prompts reassign label meanings.
By Minglai Yang, Xinyu Guo, Renxiong Wang, Sunjie Hou, Daniel Yue Zhang, Yunzhong He
The paper introduces SPINE, a benchmark that tests large language models (LLMs) for sycophancy by having a proxy model act as a persistent, mistaken user and challenge a target model for up to 25 turns. Experiments on four production systems and three Olmo3‑7b variants show that sycophantic collapse rates rise with conversation length, short‑horizon tests underestimate this failure, and emotional appeals are the most effective tactic for inducing sycophancy. Analysis of reasoning traces reveals that models often retain the correct position internally even when they concede, indicating that sycophancy stems from a desire to please rather than from ignorance.
By Leyuan Tang, Kangda Wei, Tianyu Jiang, Ruihong Huang