arXiv AI
FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks
Read the original on arXiv AI →The Flow has not summarised this story yet — read it at arXiv AI.
Related stories
arXiv AI
Jul 28
Same Question, Different Answers: Evaluating LLM Reliability Beyond Accuracy
arXiv:2607. 22554v1 Announce Type: new Abstract: Large language models (LLMs) often achieve strong accuracy on benchmarks, yet it remains unclear how reliably they apply this knowledge when the same question is phrased in different but equivalent ways.
arXiv AI
Sep 21
Hallucination-R1: Robustness-Oriented Paraphrase Generation for Factual Consistency
arXiv:2609.21227v1 Announce Type: cross Abstract: Factual hallucination is commonly defined by incorrect factual outputs. We study a paraphrase-induced hallucination setting, where a model answers a...
arXiv AI
Sep 15
How Semantically Stable Are LLM Refusals? Measuring Confusion in Local Safety Boundaries
The paper introduces Semantic Confusion to assess how consistently large language models refuse similar prompts. It presents ParaGuard, a 10k‑prompt corpus of controlled paraphrase clusters, and proposes three token‑level metrics—Confusion Index, Confusion Rate, and Confusion Depth—to measure contradictory refusal decisions across meaning‑preserving paraphrases. Experiments show that global false rejection rates can mask local inconsistencies, revealing that refusal evaluation must consider both frequency and consistency across nearby paraphrases.
arXiv AI
Aug 25
POOL: Propagated Uncertainty Over Lookalikes
arXiv:2608.23086v1 Announce Type: new Abstract: Black-box large language models need confidence scores that can separate likely-correct from likely-incorrect outputs, enabling systems to prioritize h...
arXiv AI
Jun 30
Does Verbose Chain-of-Thought Really Help? In-Distribution Evidence that Content, Not Length, Matters
arXiv:2606. 30128v1 Announce Type: new Abstract: Chain-of-thought (CoT) prompting improves LLM reasoning, but the source is contested: do the intermediate steps help because they carry useful semantic content, or because conditioning on more tokens buys extra computation before the model commits to an answer?
arXiv Computation and Language
Aug 25
Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA
arXiv:2608.22856v1 Announce Type: cross Abstract: A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval pol...