The paper introduces CarryOnBench, an interactive benchmark that tests whether large language models can revise their interpretation of user intent and recover utility while staying safe in multi‑turn conversations. Using 398 harmful‑looking queries with benign intents, the benchmark simulates 5,970 conversations across 14 models, evaluating both intent‑aligned utility and safety with a new metric called Ben‑Util. Results show that models often withhold information due to misinterpretation, but most can recover with clarifications, revealing failure modes such as unsafe and redundant recovery that single‑turn tests miss.
By Mingqian Zheng, Malia Morgan, Liwei Jiang, Carolyn Rose, Maarten Sap
The paper introduces a counterfactually anchored evidence attribution approach for multi‑turn large language model safety failures. It presents a new dataset of 1,762 conversations, including adversarial, benign twins, and high‑risk vocabulary variants, and trains a lightweight hierarchical model that accurately predicts safety violations and attributes them to specific user turns and token spans. The model achieves high detection performance (F1 = 0.988) and significantly reduces adversarial confidence when top‑attributed tokens are removed, while maintaining low false‑positive rates on benign conversations.
By Srinivasan Subramanian, Kazi Aminul Islam, Md. Abdullah Al Hafiz Khan
arXiv:2602. 16346v4 Announce Type: replace-cross Abstract: LLM-based agents execute real-world workflows via tools and memory.
By Nivya Talokar, Ayush K Tarun, Murari Mandal, Maksym Andriushchenko, Antoine Bosselut
The paper investigates how conversational safety in language models degrades over extended, adversarial interactions. By testing three instruction‑tuned models with persistent adversarial users across up to 101 turns, the study finds that safe‑response rates drop sharply from 85–100% at the first turn to 15–44% by the end. This demonstrates that strong single‑turn safety does not guarantee continued safety in long conversations.
By Parisa Salmani, Peter R. Lewis
Large language models (LLMs) are increasingly used across diverse tasks in K-12 education, yet existing safety evaluations rarely examine how harmful or inappropriate content appears in interactions between LLMs and students or teachers. To address this, we present EduZone, an evaluation framework for LLM safety across diverse educational scenarios.
arXiv:2609.38389v1 Announce Type: cross
Abstract: Large language models (LLMs) remain vulnerable to adversarial attacks that circumvent safety alignment to elicit harmful outputs. It remains unclear...
By Yelyzaveta (Lisa), Husieva, Lauren Alvarez
PASTABench introduces a benchmark of 1,139 multi-turn trajectories to evaluate proactive safety monitoring in large language models. It formalizes three dimensions of intervention—whether, when, and what risk—to address gaps in step-level isolation and post-hoc trajectory assessment. The study finds that proactive intervention is largely unsolved, with the best model achieving only 40.74% optimal-timing interventions, and reveals that smaller models’ safety scores are often driven by lexical overfitting rather than true risk comprehension.
By Jiapeng Sun, Yujin Zhou, Han Zhu, Pengcheng Wen, Jiayi Zhou, Sirui Han, Yike Guo
arXiv:2607. 19361v1 Announce Type: cross Abstract: Most safety guardrails for large language models (LLMs) evaluate each prompt-response pair in isolation, which misses failures that arise only over a dialogue as benign turns compose into harm.
By Sanjay Mishra, Divya Chukkapalli, Ganesh R. Naik
arXiv:2608.21775v1 Announce Type: new
Abstract: Large Language Models (LLMs) are increasingly deployed in real-world applications, yet they remain vulnerable to generating harmful content. From adver...
By Afshin Orojlooyjadid, Hitesh Patel
arXiv:2606. 09890v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as autonomous agents capable of executing multi-step action trajectories toward a given objective.
By Hainiu Xu, Italo Luis da Silva, Jiangnan Ye, Yuhao Wang, Wei Liu, Linyi Yang, Jonathan Richard Schwarz, Nicola Paoletti, Yulan He, Hanqi Yan
arXiv:2503. 15560v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly vulnerable to sophisticated multi-turn manipulation attacks, where adversaries strategically build context through seemingly benign conversational turns to circumvent safety measures and elicit harmful or unauthorized responses.
By Prashant Kulkarni, Assaf Namer
arXiv:2607. 26820v1 Announce Type: new Abstract: As large language models (LLMs) evolve from standalone assistants into autonomous agents, ensuring their safety requires shifting beyond pointwise risk assessment to understand how risks emerge and unfold over long-horizon trajectories.
By Shi Lin, Peng Qian, Dinghao Liu, Renjie Sun, Sifan Wu, Dezhang Kong, Chenpei Wang, Xun Wang