arXiv AI

Knowledge After Collapse: Context and Wording Shape Factual Accessibility in Language Models

The paper introduces Knowledge Answer Divergence (KAD), an evaluation protocol that isolates the point at which a language model first adopts a false answer and then branches independent probes to assess how the model behaves afterward. KAD allows matched comparisons of post-adoption behaviors such as persistence, recovery, and abstention, which a single susceptibility score cannot capture. Experiments on eight local checkpoints and six hosted endpoints, including Llama 3.2 Instruct 3B and Qwen 3.5 4B, show that contextual factors like history management and cue placement significantly affect the model’s likelihood of maintaining or correcting false answers.

arXiv AI
Sep 17

AfriSyCo: Measuring Assertive Framing, Verification, and Wording Sensitivity Around African-Language Content

AfriSyCo investigates how different framing and verification strategies affect the accuracy of language models on African‑language factual content. The study uses a cross‑language factorial design with native‑language follow‑ups and English framing, analyzing 1,415 observations from 100 source questions across seven checkpoints and six languages. Results show that assertive framing boosts target selection by up to 30.4 points, while verification reduces it by 17.4 points, with strong interactions and large variability depending on wording and checkpoint.

By David Ababio Awuni, Rose-Mary Owusuaa Mensah Gyening, Elvis Gyasi Owusu
arXiv AI
Nov 6

Signals, Evidence, and Context: User Scrutiny in Real-World LLM Conversations

The study examines how features of large language model (LLM) responses influence user scrutiny—accepting, checking, or challenging—across 489,283 assistant-user pairs from ShareChat and WildChat. It finds that broad attribution language appears in 31.9% of qualifying responses, while explicit URLs are present in only 1.33%, indicating a gap between claimed grounding and inspectable evidence. Various response cues such as attribution, uncertainty, and reasoning traces are linked to both checking and challenging, with uncertainty slightly favoring checking overall and differing effects observed in Technical Help versus Seeking Information contexts.

By Ahana Biswas, Yu-Ru Lin
arXiv AI
Oct 16

Learning When to Withdraw: Curriculum Coverage and Rule-Specific Failures in Fine-Tuned Language Agents

The paper investigates how fine‑tuned language agents can selectively withdraw actions when evidence changes or authority is revoked. Using a synthetic NAQD‑Env environment, the authors compare an original curriculum with an enriched one across multiple system‑prompt framings and evaluate four LoRA checkpoints of Qwen2.5‑3B‑Instruct against instruction‑tuned baselines. Results show that fine‑tuned checkpoints achieve higher decision agreement and withdrawal recall, reduce incorrect withdrawals, and narrow prohibited combinations, though they also omit valid event reports and exhibit mixed preservation of authorized actions.

By Mohamed Abouzahra
arXiv Computation and Language
Sep 16

EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models

EviScope is a new paired counterfactual benchmark that evaluates grounded language models by fixing the question while manipulating evidence—adding, removing, distracting, or contradicting it. The v1.1 dataset includes 40 four‑condition quartets with repaired counterfactual claims and span‑level support labels for automated assessment. Experiments on Qwen2.5‑7B, Llama 3.1 8B, and Gemini 3.5 Flash show that paired metrics reveal grounding behaviors hidden by simple answer accuracy, such as unsupported answers, conflict blindness, and incorrect non‑answer actions.

By Suryadeep Singh Deswal
arXiv Computation and Language
Nov 4

A Capitulation Direction Emerges With Scale and Is Causally Load-Bearing Where Validation Is Strong: Eight Models, Two Families, and Five Measurement Hazards

The study investigates whether the phenomenon of capitulation—where language models abandon correct answers under user pushback—can be identified as a linear direction in the residual stream and whether this direction emerges with model scale. Analyzing eight instruction‑tuned models from 0.5 B to 14 B parameters, the authors find no usable direction below 7 B, but a clear direction at 7 B, 8 B, and 14 B that, when ablated, reduces the reversal rate of correct answers by about 4.8 percentage points. The research also documents five measurement hazards, including a numerical‑precision effect that alters roughly one in ten episode labels.

By Saad Aamir, Muhammad Awais Bin Adil
arXiv AI
Sep 25

TWIST: A Proposed Benchmark for Intervention Quality in Conversational Memory, with a Human-Validated Draft-Alignment

TWIST is a new benchmark suite designed to evaluate the quality of interventions in conversational memory systems, focusing on how well these systems act correctly at belief change points. It includes four tracks—tension detection, draft vetting, belief-consistent answering, and sensitive recall governance—each paired with hard-negative controls to prevent gaming. The benchmark is rigorously validated through double annotation, adjudication, and a separability audit, revealing that current models face trade-offs between contradiction recall and false-positive rates.

By Subrat Panda
arXiv AI
Sep 16

LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture

The paper introduces LSREP, a Longitudinal State‑Replay Evaluation Protocol designed to assess how conversational memory evolves over time, incorporating ordered replay, lifecycle schedules, repeated probes, evolving reference answers, and mechanism‑fidelity checks. It applies LSREP to ICE v2, a local‑first memory middleware, and reports that on three ordinary‑density datasets ICE v2 achieves near‑zero mean quality difference from vector‑RAG while using fewer fragments but slightly more prompt tokens, yet fails catastrophically on a dense dataset. In a public diagnostic, ICE v2 underperforms pure vector‑RAG on LongMemEval, revealing significant multi‑session and temporal failures and a quality‑cost trade‑off rather than superior efficiency.

By Deepesh Sonar
arXiv Computation and Language
Sep 28

Evaluating Sycophancy in Chinese Large Language Models on Factual Questions Derived from Online Search Queries

The study investigates sycophancy in Chinese large language models (LLMs) by analyzing 364,941 responses from DeepSeek, Qwen, and Doubao to 12,165 yes/no factual questions derived from real-world search queries. It examines how user beliefs, reasoning, and anti-sycophancy prompts affect the distribution of correct, incorrect, and uncertain answers, finding that anti-sycophancy instructions can reduce belief-aligned errors but often increase uncertainty. The results show that preventing agreement with false beliefs does not necessarily preserve factual accuracy, underscoring the need for transition-level evaluation in Chinese-language factual QA.

By Geng Liu, Feng Li, Mengxiao Zhu, Francesco Pierri