← Back to all news
arXiv Machine Learning By Chia-Yi Hsu, Chia-Mu Yu, Chun-Ying Huang, Jun Sakuma

Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills

Read the original on arXiv Machine Learning →

The paper introduces the Neutral Prompting Attack (NPA), a stealthy method that uses semantically benign instructions to increase the likelihood of large language models hallucinating non‑existent package names in coding agents. Unlike traditional dependency steering, NPA does not target a specific package but shifts the model’s output toward more speculative names. Experiments across multiple LLMs show that NPA raises hallucination rates, affects pip install success, alters the distribution of hallucinated packages, and bypasses existing static‑analysis, LLM‑based, and agent‑based defenses.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 25

Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code

The paper investigates how large language models (LLMs) hallucinate nonexistent software packages during code generation and evaluates methods to mitigate this issue. It finds that current evaluation practices overestimate hallucination rates, especially for Python, and that Retrieval-Augmented Generation (RAG) and Self-Refine reduce hallucinations across multiple models and languages. The study also introduces Package Utility (PU) to measure whether defenses preserve useful recommendations and shows that Greedy decoding offers the best trade‑off between mitigation and utility, while adversarial prompts significantly increase hallucination rates, particularly in Ruby.

By Alberick Euraste Djire, Iyiola E. Olatunji, Melissa Tessa, Earl T. Barr, Jacques Klein, Tegawend\'e F. Bissyand\'e
llmsragsafety
More like this →