arXiv AI

Beyond Prompt Engineering: A Systematic Analysis of Prompt Lexical Sensitivity and Its Impacts on Quality

The paper investigates how small lexical changes in prompts can cause large performance swings in large language models. Using a dataset of 132,000 prompt variants, the authors uncover a scaling law linking higher average task performance to lower variance and greater robustness. They identify domain-specific terminology and explicit action directives as key linguistic factors that stabilize prompts, and propose an automated Prompt-Refining Agent that reduces performance variance by 40.7% in code generation while maintaining or improving mean performance.

arXiv AI

Aug 20

Evaluating and Explaining Prompt Sensitivity of LLMs Using Interactions

The paper introduces a fine-grained method called interactions to analyze prompt sensitivity in large language models (LLMs). By decomposing output scores into nonlinear interactions, the authors show that subtle prompt changes can destabilize these interactions even when overall outputs stay unchanged. They propose an Interaction-based Prompt Sensitivity (IPS) metric and use it to evaluate 50 open-source LLMs, finding that supervised fine‑tuning, larger model scales, dense architectures, and few‑shot learning all reduce prompt sensitivity, primarily by stabilizing low‑order interactions.

By Ruiyang Qin, Qingzhuo Wang, Tian Wang, Zhihua Wei, Wen Shen

llms fine-tuning

arXiv AI

Jul 17

Automatically Evolving Prompt Guidelines for Task-Specific Optimization

arXiv:2607. 14105v1 Announce Type: cross Abstract: For Large Language Models to reliably answer user queries, users must clearly specify requirements, context, and constraints.

By Cedric Richter, Salah Ghamizi, Mike Papadakis

llms nlp benchmarks

arXiv AI

Jun 9

Knowing How to Edit: Reliable Evaluation Signals for Diagnosing and Optimizing Prompts at Query Level

arXiv:2511. 19829v3 Announce Type: replace Abstract: Prompt optimization has become a central mechanism for eliciting strong performance from LLMs, and recent work has made substantial progress by proposing diverse prompt evaluation metrics and optimization strategies.

By Ke Chen, Yifeng Wang, Hassan Almosapeeh, Haohan Wang

llms benchmarks

arXiv AI

5d ago

PromptResponse: Optimizing Prompts for LLM Coding Tasks

arXiv:2608.21074v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used in research workflows and software development pipelines, yet their output remains sensitive to in...

By Erik Thureck, Robert K\"uhnen, Tim Jacobowitz

llms benchmarks safety

arXiv Machine Learning

1d ago

$p1$: Better Prompt Optimization with Fewer Prompts

arXiv:2604.08801v2 Announce Type: replace Abstract: Prompt optimization improves language models without updating their weights by searching for a better system prompt, but its effectiveness varies w...

By Zhaolin Gao (Sid), Yu (Sid), Wang, Bo Liu, Thorsten Joachims, Kiant\'e Brantley, Wen Sun

llms benchmarks

arXiv Machine Learning

Jun 24

Stabilizing Black-Box Prompt Optimization with Textual Regularization and Signal Aggregation

arXiv:2507. 09839v2 Announce Type: replace Abstract: An increasing number of NLP applications interact with large language models (LLMs) through black-box APIs, making prompt engineering critical for controlling model behavior.

By MohammadReza Davari, Utkarsh Garg, Weixin Cai, Eugene Belilovsky

llms nlp

arXiv AI

Jul 1

Shared Lexical Task Representations Explain Behavioral Variability In LLMs

arXiv:2604. 22027v2 Announce Type: replace-cross Abstract: One of the most common complaints about large language models (LLMs) is their prompt sensitivity -- that is, the fact that their ability to perform a task or provide a correct answer to a question can depend unpredictably on the way the question is posed.

By Zhuonan Yang, Jacob Xiaochen Li, Francisco Piedrahita Velez, Eric Todd, David Bau, Michael L. Littman, Stephen H. Bach, Ellie Pavlick

llms

Hugging Face Trending Papers

Aug 13

Prompts in the Wild: A Large Analyzed Collection of Transactional Prompts in Code

The behavior of contemporary generative Large Language Models (LLMs) is directly shaped by prompts, unstructured texts that describe the desired output and model behavior. In this paper we argue that prompts are linguistic objects that merit investigation in their own right.

llms

arXiv AI

Jun 11

APEX: Automated Prompt Engineering eXpert with Dynamic Data Selection

arXiv:2606. 11459v1 Announce Type: cross Abstract: Large Language Models are highly sensitive to prompt formulation, necessitating automatic prompt optimization to unlock their full potential.

By Fei Wang, Si Si, Cho-Jui Hsieh, Inderjit S. Dhillon

llms benchmarks

arXiv AI

Jun 2

A Unified Evaluation-Instructed Framework for Query-Dependent Prompt Optimization

arXiv:2511. 19829v2 Announce Type: replace Abstract: Most prompt-optimization methods refine a single static template, making them ineffective in complex and dynamic user scenarios.

By Ke Chen, Yifeng Wang, Hassan Almosapeeh, Haohan Wang

arXiv Machine Learning

Aug 6

From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMs

arXiv:2601. 03808v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have achieved notable performance in code synthesis; however, data-aware augmentation remains a limiting factor, handled via heuristic design or brute-force approaches.

By Usha Shrestha, Dmitry Ignatov, Radu Timofte

llms reinforcement-learning fine-tuning benchmarks safety

arXiv Machine Learning

Jun 17

Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generation

arXiv:2602. 03045v2 Announce Type: replace Abstract: Large language models have recently enabled text-to-CAD systems that synthesize parametric CAD programs (e.

By Bo Yuan, Zelin Zhao, Petr Molodyk, Bin Hu, Yongxin Chen

llms agents fine-tuning

Related stories