← Back to all news
arXiv Machine Learning By Chia-Yi Hsu, Chia-Mu Yu, Chun-Ying Huang, Jun Sakuma

Harmless Yet Harmful: Neutral Prompting Attacks for Stealthy Hallucination Steering in Agent Skills

Read the original on arXiv Machine Learning →

The paper introduces the Neutral Prompting Attack (NPA), a stealthy method that uses semantically benign instructions to increase the likelihood of large language models hallucinating non‑existent package names in coding agents. Unlike traditional dependency steering, NPA does not target a specific package but shifts the model’s output toward more speculative names. Experiments across multiple LLMs show that NPA raises hallucination rates, affects pip install success, alters the distribution of hallucinated packages, and bypasses existing static‑analysis, LLM‑based, and agent‑based defenses.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv Machine Learning.

Related stories

arXiv AI
Aug 25

Evaluating Inference-Time Defenses Against Package Hallucination in LLM-Generated Code

The paper investigates how large language models (LLMs) hallucinate nonexistent software packages during code generation and evaluates methods to mitigate this issue. It finds that current evaluation practices overestimate hallucination rates, especially for Python, and that Retrieval-Augmented Generation (RAG) and Self-Refine reduce hallucinations across multiple models and languages. The study also introduces Package Utility (PU) to measure whether defenses preserve useful recommendations and shows that Greedy decoding offers the best trade‑off between mitigation and utility, while adversarial prompts significantly increase hallucination rates, particularly in Ruby.

By Alberick Euraste Djire, Iyiola E. Olatunji, Melissa Tessa, Earl T. Barr, Jacques Klein, Tegawend\'e F. Bissyand\'e
llmsragsafety
More like this →
arXiv AI
2d ago

NameTrap: Steering AI Coding Agents Toward Security Vulnerabilities via Variable Names

The paper introduces NameTrap, a source‑code poisoning attack that renames identifiers in third‑party Python files to steer AI coding agents toward specific Common Weakness Enumeration (CWE) vulnerabilities. By iteratively aligning variable names with CWE‑specific conventions, the attack achieves success rates ranging from 22.5% to 75.0% across six agents and four vulnerability categories, outperforming random and neutral renaming strategies. The poisoned files contain no explicit instructions or vulnerable statements, and existing AI‑based and static security analyzers fail to detect the induced weaknesses.

By Chieh-Yuan Chang, Guan-Wei Chen, Chia-Yi Hsu, Chih Yang, Chia-Mu Yu, Chun-Ying Huang
llmsagentsrobotics
More like this →