arXiv AI

When File Ingestion Misleads Language Models: Measuring and Mitigating Divergences in Office Documents

The paper investigates how large language models (LLMs) misinterpret office documents due to differences between the text extracted from Office Open XML (OOXML) files and what humans actually see. By analyzing common extractors, the authors identify 19 hidden OOXML channels and 12 content omissions, and demonstrate that these gaps enable five attacks that mislead major chat and coding assistants. They propose Prism, a lightweight defense that reconstructs the human view from OOXML before feeding text to the model, which generalizes to unseen attack payloads.

arXiv AI
Jun 17

Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks

arXiv:2510. 01359v2 Announce Type: replace-cross Abstract: Code-capable large language model (LLM) agents are embedded in software engineering workflows where they can read, write, and execute code, raising "jailbreak" stakes beyond text-only settings.

By Shoumik Saha, Jifan Chen, Sam Mayers, Sanjay Krishna Gouda, Zijian Wang, Varun Kumar
arXiv Computation and Language
Aug 27

A Layered Security Framework Against Prompt Injection in RAG-Based Chatbots

The paper introduces a three‑layer security framework designed to protect retrieval‑augmented generation (RAG) chatbots from both direct and indirect prompt injection attacks. Layer 1 filters user input with rule‑based patterns and a semantic anomaly classifier; Layer 2 enforces a provenance‑based instruction hierarchy during context assembly; Layer 3 audits model output with a policy rule engine and semantic drift detector. Evaluations on GPT‑4o, Llama 3, and Mistral 7B demonstrate a reduction in attack success rate from 71.4 % to 11.3 %, outperforming existing single‑layer defenses while keeping false positives low and latency acceptable.

By Gulshan Saleem, Nisar Ahmed, Muhammad Imran Zaman, Ali Hassan, Umar Mujahid
arXiv AI
Aug 26

Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors

The paper introduces Semantic Overlays, a steering technique that adds non‑textual annotations to a language model’s input by applying learned adapters at specific prefill positions. These overlays create an out‑of‑band channel that encodes span identity and complex semantics, enabling the model to interpret marked text differently—such as rewriting code in a specified language or ignoring executable instructions. Experiments show that Semantic Overlays dramatically reduce prompt‑injection success rates while preserving model utility and readability of marked spans.

By Joshua Penman
arXiv AI
Jun 2

AgentRedBench: Dynamic Redteaming and Integration-Aware Defense for LLM Agents over SaaS Integrations

arXiv:2606. 02240v1 Announce Type: cross Abstract: Indirect prompt injection in tool-use agents is a concrete production threat: LLM agents read from integrations (third-party services such as Gmail, Salesforce, or Jira accessed through tool calls) whose response content the user neither writes nor controls.

By Hiskias Dingeto, Will Leeney
arXiv Machine Learning
Jul 30

RAGuard: A Layered Defense Framework for Retrieval-Augmented Generation Systems Against Data Poisoning

arXiv:2607. 26339v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) systems ground large language models (LLMs) in external corpora, but this reliance exposes them to corpus poisoning: maliciously injected passages that manipulate retrieved evidence.

By Pushkal Kumar, Tucker Nielson, Tanish Kolhe, Shubham Zala, Vincent Li
arXiv AI
Oct 28

When AI Agrees Too Much: Manipulating Automated Code Review via Bias Injection

The paper investigates how large language model (LLM)–based automated code review (ACR) systems can be manipulated through contextual bias injection, a framing effect that lets presentation override content. A large‑scale study across six LLMs confirms the framing effect in vulnerability detection, and a realistic experiment on 33 CVEs in 20 projects shows that a novel LLM‑assisted refinement attack succeeds in 97% of cases, while a template‑based attack fails. The results underscore the vulnerability of real‑world ACR pipelines and the need for human oversight and trust in the development process.

By Georgios Alexopoulos, Nikolaos Alexopoulos, Diomidis Spinellis, Dimitris Mitropoulos