The paper introduces a new metric for assessing model coherence by measuring contradictions that arise when resampling answers to a carefully curated set of 175 questions. It highlights two overlooked causes of incoherence—ambiguity and indifference—and demonstrates that narrow finetunes often score poorly, revealing issues such as identity conflation, introspection failures, and rationalizations. The study argues that these pathologies may limit the usefulness of such models for understanding coherent misaligned behavior.
arXiv:2603. 28371v2 Announce Type: replace-cross Abstract: When an agent can articulate why something works, we typically take this as evidence of genuine understanding.
The paper introduces $A^2E^2$, a diagnostic framework that decomposes confusion in fine‑grained aircraft detection into four distinct, measurable sources: affinity, heterogeneity, contested, and collapsed. By attributing each source to either aleatoric or epistemic uncertainty and within‑ or between‑class distinctions, the method provides actionable remedies and experimentally verifies that targeted interventions reduce the identified source without affecting irreducible ones. This turns passive confusion matrices into a concrete, validatable diagnosis that can guide model improvement.
arXiv:2503. 13445v3 Announce Type: replace-cross Abstract: When asked to explain their decisions, LLMs can often give explanations which sound plausible to humans.
The paper investigates whether the tokens highlighted by large language models (LLMs) as key to their predictions are behaviorally consistent when those tokens are manipulated. Using sufficiency, comprehensiveness, and counterfactuality metrics, the authors evaluate closed‑source GPT‑4o‑mini and open‑source models Gemma3, Granite8B, and DeepSeek on IMDB and Steam review datasets. Results show that GPT‑4o‑mini consistently follows expected patterns, Gemma3 and Granite8B maintain sufficiency but falter under stricter interventions, and DeepSeek variants exhibit structural inconsistencies, indicating that explanation reliability varies across model families and sizes.
Consistency training encourages a model to produce similar outputs across related inputs or sampling procedures. Such methods are simple, scalable, and largely label-free, but their effects on model alignment remain poorly understood.
arXiv:2607. 19367v1 Announce Type: new Abstract: Calibration is the primary criterion for evaluating LLM confidence, but it is insufficient: it admits trivially incoherent estimators, depends on the evaluation distribution, and does not test the extent to which the estimation can be interpreted as a consistent, underlying probability function.
arXiv:2605.30381v2 Announce Type: replace-cross Abstract: When a language model is fine-tuned to produce systematically incorrect responses, does this training leave a structured, linearly recoverabl...
The study examines how different editorial framings in prompts influence large language models’ statistical analysis reports. Using a 4×4 factorial design, researchers found that certain framings—particularly brutally critical prompts on genuine effects and significance-seeking prompts on underpowered nulls—led to factual misrepresentations. Tone shifts were more widespread, with critical framing inducing defensive language across all data patterns, while a confound in the data largely prevented both factual and tonal distortions.
arXiv:2606. 03810v1 Announce Type: cross Abstract: Consistency training encourages a model to produce similar outputs across related inputs or sampling procedures.
arXiv:2607. 26389v1 Announce Type: cross Abstract: Fine-tuning a language model on data containing a narrow flaw, such as insecure code or incorrect mathematical answers, can cause broad misalignment through a mechanism that remains debated.
arXiv:2607. 28576v1 Announce Type: cross Abstract: Methods that make a language model plan, criticise and rewrite its own answer, reflect on mistakes, pick the best of several attempts, or debate with copies of itself nearly all make it generate far more text than a single chain of thought.
arXiv:2604.12128v2 Announce Type: replace Abstract: Do language models struggle with self-reference itself, or with structures where propagating truth values along references cannot settle a value in...