arXiv:2607.00711v2 Announce Type: cross Abstract: Large Language Models have emerged as programming assistants. However, the efficacy of code generation is constrained by the quality of input require...
arXiv:2607. 04436v1 Announce Type: cross Abstract: Natural language requirements (NLRs) are essential for bridging communication gaps among diverse stakeholders in software development.
The paper introduces ReqClarifyBench, an interactive benchmark for evaluating how well large language models (LLMs) can clarify vague user intent into clear requirements in repository-level software development. It includes 341 tasks across game, miniapp, and web development, each with a vague intent, a human-validated requirement document, and code evaluation artifacts, and uses LLM-based user simulators with diverse personas. The study evaluates six LLMs on clarification effectiveness and requirement quality, finding that stronger clarification does not always yield better documents, performance varies by persona, and coding-agent harnesses do not improve document quality over base LLMs.
arXiv:2602. 22456v2 Announce Type: replace-cross Abstract: Requirements are inherently interconnected through various types of dependencies.
The paper introduces a leakage‑aware framework for conversational task disambiguation over tabular data, defining ambiguous verifiable tasks and separating the agent into an asking policy and a solution policy. It presents AmbiTab, a benchmark suite that unifies six ambiguous datasets under a common representation, enabling evaluation of clarification strategies and oracle leakage. The authors train an asking policy with reinforcement learning, achieving improved disambiguation metrics across all datasets and higher task success on five of them, while providing diagnostics to measure oracle leakage during training.
HoarePrompt is a new method that applies program verification concepts to natural language requirements, using large language models to generate step‑by‑step natural language descriptions of program states. It incorporates a few‑shot k‑induction technique to handle loops and then evaluates whether the annotated program satisfies the requirements. On the CoCoClaNeL dataset, HoarePrompt raises the Matthews correlation coefficient by 61% over zero‑shot chain‑of‑thought prompts and by 106% over test‑generation classifiers, with the inductive reasoning component adding a 26% MCC improvement.
arXiv:2606. 17164v1 Announce Type: cross Abstract: Prompting has become the primary interface between humans and generative AI, yet many natural language prompts remain fragile: roles, goals, constraints, and expected outputs are often buried in prose or left implicit.
arXiv:2608. 08700v1 Announce Type: new Abstract: Reliable evaluation of tool routing is critical as Large Language Models increasingly operate as autonomous agents.
The paper introduces Trustworthy RAG, an evaluation agent designed to detect misinformation and knowledge poisoning in Retrieval-Augmented Generation systems. It combines natural language inference verification, a five-signal poison detector, and a weighted Trust Index to assess the reliability of retrieved content. Experiments on multiple LLMs show high accuracy and precision, with the agent effectively blocking unsafe advice in a secure-coding assistant scenario.
The paper investigates whether large language models (LLMs) can infer the capability requirements of a user query before generating a response. Using the TACIT framework, the authors decompose these requirements into eight classes across three axes—Source, Transformation, and World Effect—and train linear probes on hidden states from four open-weight LLM families. Their results show that these capability structures are linearly decodable with high accuracy, yet the models struggle to express the same information in natural language, a phenomenon termed "structured but silent."
arXiv:2606. 26101v1 Announce Type: cross Abstract: Reliable evaluation of large language models should separate supported answering from unsupported guessing without conflating either with data contamination, prompt idiosyncrasy, or generic refusal behavior.
arXiv:2609.06052v1 Announce Type: cross Abstract: Autonomous agent systems increasingly depend on reusable skill abstractions for consolidating experiential knowledge and domain expertise. These arti...