arXiv AI September 17, 2026 By David Ababio Awuni, Rose-Mary Owusuaa Mensah Gyening, Elvis Gyasi Owusu

AfriSyCo: Measuring Assertive Framing, Verification, and Wording Sensitivity Around African-Language Content

AfriSyCo investigates how different framing and verification strategies affect the accuracy of language models on African‑language factual content. The study uses a cross‑language factorial design with native‑language follow‑ups and English framing, analyzing 1,415 observations from 100 source questions across seven checkpoints and six languages. Results show that assertive framing boosts target selection by up to 30.4 points, while verification reduces it by 17.4 points, with strong interactions and large variability depending on wording and checkpoint.

Machine-generated by The Flow from the publisher's headline and feed description — not written or checked by a human. The full article lives at arXiv AI.

arXiv Machine Learning

Sep 30

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

arXiv:2609.37616v1 Announce Type: new Abstract: Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on th...

By Abhinav Rajeev Kumar (Lossfunk), Paras Chopra (Lossfunk)

llms benchmarks safety

arXiv AI

Aug 25

Evaluation Awareness in Language Models: Representation, Verbalization, and Control

arXiv:2608.21766v1 Announce Type: cross Abstract: Both capability and safety benchmarks rest upon the assumption that the behavior of language models undergoing a test is informative about their beha...

By Farzaneh Heidari, Amin Memarian, Guillaume Rabusseau

llms fine-tuning benchmarks safety

arXiv Computation and Language

Sep 16

EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models

EviScope is a new paired counterfactual benchmark that evaluates grounded language models by fixing the question while manipulating evidence—adding, removing, distracting, or contradicting it. The v1.1 dataset includes 40 four‑condition quartets with repaired counterfactual claims and span‑level support labels for automated assessment. Experiments on Qwen2.5‑7B, Llama 3.1 8B, and Gemini 3.5 Flash show that paired metrics reveal grounding behaviors hidden by simple answer accuracy, such as unsupported answers, conflict blindness, and incorrect non‑answer actions.

By Suryadeep Singh Deswal

llms rag benchmarks

arXiv AI

Sep 25

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

PROOF is a benchmark that profiles the reliability of object-level facts in instruction-tuned language models by converting a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts across 101 classes, 392 properties, and 14 domains. Each question includes an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations, with 1,849 questions designed as no-correct-option traps. The study evaluates 18 open-weight model deployments on 166,374 prompts, revealing wide variability in factual accuracy, sensitivity to wording changes, and the impact of decoder perturbations.

By Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha, Stepan Ukolov, Valeriia Kuschenko, Alexander Evseev, Sergey Bolovtsov

llms benchmarks safety

arXiv AI

Jun 12

Prefill Awareness in Large Language Models

arXiv:2606. 12747v1 Announce Type: new Abstract: Safety-relevant studies of language models, including alignment and jailbreaking evaluations and AI control protocols, often rely on prefilling model outputs.

By Andy Wang, Parv Mahajan, David Demitri Africa, Alexandra Souly, Jordan Taylor, Robert Kirk

llms agents benchmarks safety

arXiv Machine Learning

Jul 30

Same Evidence, Different Target: Decoding How Diagnostic Evidence Bears on Causal Questions from Language-Model States

arXiv:2607. 26929v1 Announce Type: cross Abstract: The same diagnostic result can support or challenge one causal claim yet fail to address another when the claims concern different populations, outcomes, estimands, pathways, or identifying assumptions.

By Weiyi Kong, Zhuoran Li

llms benchmarks

One email a morning, machine-written

Related stories

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

Evaluation Awareness in Language Models: Representation, Verbalization, and Control

EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

Prefill Awareness in Large Language Models

Same Evidence, Different Target: Decoding How Diagnostic Evidence Bears on Causal Questions from Language-Model States