arXiv:2607. 22554v1 Announce Type: new Abstract: Large language models (LLMs) often achieve strong accuracy on benchmarks, yet it remains unclear how reliably they apply this knowledge when the same question is phrased in different but equivalent ways.
arXiv:2609.21227v1 Announce Type: cross Abstract: Factual hallucination is commonly defined by incorrect factual outputs. We study a paraphrase-induced hallucination setting, where a model answers a...
The paper introduces Semantic Confusion to assess how consistently large language models refuse similar prompts. It presents ParaGuard, a 10k‑prompt corpus of controlled paraphrase clusters, and proposes three token‑level metrics—Confusion Index, Confusion Rate, and Confusion Depth—to measure contradictory refusal decisions across meaning‑preserving paraphrases. Experiments show that global false rejection rates can mask local inconsistencies, revealing that refusal evaluation must consider both frequency and consistency across nearby paraphrases.
arXiv:2608.23086v1 Announce Type: new Abstract: Black-box large language models need confidence scores that can separate likely-correct from likely-incorrect outputs, enabling systems to prioritize h...
arXiv:2606. 30128v1 Announce Type: new Abstract: Chain-of-thought (CoT) prompting improves LLM reasoning, but the source is contested: do the intermediate steps help because they carry useful semantic content, or because conditioning on more tokens buys extra computation before the model commits to an answer?
arXiv:2608.22856v1 Announce Type: cross Abstract: A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval pol...
arXiv:2607. 14528v1 Announce Type: cross Abstract: Large language models (LLMs) frequently contradict themselves when the surface form of a logically equivalent question changes.
arXiv:2608. 11138v1 Announce Type: cross Abstract: We propose that a model's uncertainty about a token is reflected not only in the breadth of its output distribution but also in whether a confident prediction is \emph{fragile} under perturbation of its attention pathways.
PROOF is a benchmark that profiles the reliability of object-level facts in instruction-tuned language models by converting a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts across 101 classes, 392 properties, and 14 domains. Each question includes an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations, with 1,849 questions designed as no-correct-option traps. The study evaluates 18 open-weight model deployments on 166,374 prompts, revealing wide variability in factual accuracy, sensitivity to wording changes, and the impact of decoder perturbations.
CiteVQA is a benchmark that evaluates multimodal large language models on both answering document questions and providing element‑level bounding‑box citations for the evidence used. It contains 1,897 questions across 711 PDFs in seven domains and two languages, with ground‑truth citations generated via an automated pipeline and strict quality control. The evaluation metric, Strict Attributed Accuracy (SAA), rewards a prediction only when both the answer and the cited region are correct, revealing widespread attribution hallucination in current models.
The paper introduces a benchmark of 3,600 exact‑rational word problems and 8,600 prompts that test whether language models give the same canonical answer when the same quantity is expressed in different numeric forms (decimal, fraction, percentage, number word, scientific notation, or unit‑converted). After normalizing answer syntax, canonical accuracy is high (0.969–0.996), but correctness across equivalent representations drops to 0.848–0.981, revealing that many errors stem from the evaluator’s number grammar rather than the models’ reasoning. The study also finds that representation consensus does not outperform paraphrase consensus on low‑error subsets and that certain models (e.g., Mistral Small 4) exhibit systematic unit‑conversion errors.
arXiv:2603.28301v2 Announce Type: replace Abstract: Vision-Language-Action (VLA) models achieve strong performance in robotic manipulation by leveraging pre-trained vision-language backbones. However...